How To: How can I compute group statistics by category for a large dataset.

The RevoScaleR function rxSummary() comes in very handy for this kind of thing. By default it will report group means and standard deviations as well as cell counts and number of valid observations, if you specify a formula that includes interaction terms. In the below examples we use the shipped dataset 'CensusWorkers.xdf'. 

testDataDir <- file.path(rxGetOption("sampleDataDir")) 
rxSummary(incwage ~ sex:state, data = file.path(testDataDir, "CensusWorkers.xdf")) 

Number of valid observations: 351121 
Number of missing observations: 0 
Name Mean StdDev Min Max ValidObs 
incwage:sex:state 35333.84 40444.54 0 354000 351121 

Statistics by category (6 categories): 

Category sex state Means StdDev 
incwage for sex=Male, state=Connecticut Male Connecticut 55002.00 67742.93 
incwage for sex=Female, state=Connecticut Female Connecticut 32605.03 34426.27 
incwage for sex=Male, state=Indiana Male Indiana 38325.33 36160.07 
incwage for sex=Female, state=Indiana Female Indiana 23117.71 20371.68 
incwage for sex=Male, state=Washington Male Washington 41001.14 43962.65 
incwage for sex=Female, state=Washington Female Washington 25765.03 25191.75 

Min Max ValidObs MissingObs 
0 354000 41751 0 
0 354000 37666 0 
0 314000 74221 0 
0 314000 63702 0 
0 336000 73372 0 
0 336000 60409 0 

You can use the 'rowSelection' argument to further subset your data based on specific values of some of the variables. Another useful function is the 'F()' function, which tells RevoScaleR to treat a specific variable as a factor: 

### Get information on variable types in XDF file 
rxGetInfo(file.path(testDataDir,"CensusWorkers.xdf"), getVarInfo=TRUE) 
rxSummary(incwage ~ F(age):sex, data = file.path(testDataDir,"CensusWorkers.xdf"), rowSelection = (state == "Indiana"))   
Note This is a "FAST PUBLISH" article created directly from within the Microsoft support organization. The information contained herein is provided as-is in response to emerging issues. As a result of the speed in making it available, the materials may include typographical errors and may be revised at any time without notice. See Terms of Use for other considerations.

ID článku: 3104266 - Poslední kontrola: 10/29/2015 08:28:00 - Revize: 1.0

Revolution Analytics

  • KB3104266
Váš názor