ACCF90 - Approximate accuracy program for maternal models I. Misztal. 6/2000-3/2001 ACCF90 is a program that calculates approximations to single trait direct and maternal accuracy in single- and multi-trait models. Accuracies for all traits and both effects are calculated simultaneously. The parameter file is the same as for BLUP programs (BLUP90IOD) so obtaining BLUP solutions requires only running one extra program. The renumbering program for multiple trait data does not produce incomplete parameter file so the computations requires at east one manual intervention. Rework to multiple-trait approximation will be done as time becomes available. The programs are currently running on dodo3.ads.uga.edu. Use /scr directory and create your own subdirectory for computations. Steps ===== 1. Issue commands to gain access to programs (one time) 2. Create data and pedigree files with fields separated by spaces and with spaces in IDs filled with an extra character (program cut_fill) 3. Replace IDs denoting unknown parents to 0 4. Renumber the data and pedigrees (program renummat2); animals may have IDs up to 20 characters; other effects, like mgt, need to be numeric. 5. Convert the paramater file created by renummat2 to multitrait format, possibly by changing the number of cg levels and animals of template parameter files. 6. Run program blup90iod to test parameter file and to get BLUP solutions 7. Run program accf90 with the identical parameter file to get approximate accuracies 8. Merge output of accf90 and renummat2 to match accuracies and solutions with original IDs. Option ------ Parameters of the algorithm can be controlled by adding the following line at the end of the parameter file: OPTION acc_par rounds fraction_old where rounds is the number of rounds of iteration and fraction_old specifies fraction of old solutions considered in next round solution; defaults are: rounds=20 and fraction_old =.5; The last variables must be between 0 and 1; values of 0 may result in divergence and values of close to 1 in small convergence. More rounds do not guarantee higher quality accuracies. OPTION cg 1 1 : effect number for contemporary group or herd-year-season for each trait OPTION hs 3 3 : effect number for hers by sire interaction OPTION anim 4 4 : effect number for animal or sire effect OPTION mat 5 5 : effect number for maternal or MGS effect OPTION pe 6 6 : permanent environmental effect OPTION add_residual 7 8 calculate accuracies adding variances for 7 and 8 effects to residual variance OPTION model animal model can be animal or sire OPTION type 1.0 1.0 for dairy or 0.5 for BIF OPTION acc_maxrounds 5 usually 5-10 is enough OPTION conv_crit 1d-10 OPTION parent_avg calculates parent averages Detailed instrucions and sample run =================================== ----------------------------------------------------------------------- All files create dbelow files are on dodo3.ads.uga.edu in directory /scr/ignacy/andra ----------------------------------------------------------------------- 1. Link programs so that they can be accessed without typing long names $ ln -s /home/ignacy/f90-99/acc/accf90 . $ ln -s /home/ignacy/f90-99/acc/cut_fill . $ ln -s /home/ignacy/f90-99/acc/renummat3 . $ ln -s /home/ignacy/f90-99/bin/blup90iod . 2. Cut out required fields $ cut_fill CUT_FILL 1.0 - adds space delimiters to fields in fixed column file spaces in alphanumeric fields (>= 8 characters) are replaced by % Input file? beefm.hold Output file? beefm.data give first and last column of each field, 0 0 ends 1 6 7 12 13 16 17 20 21 28 0 0 5 fields to extract length of record = 28 written 128322 records and filled 148830 spaces $ head beefm.data 000001 000000 0577 0000 C797861% 000001 000000 0645 0000 C797863% 000001 000000 0725 0000 C797859% 000001 000000 0817 0000 C797860% 000002 000000 0625 0000 C788574% 000002 000000 0642 0000 C788575% 000003 000000 0519 0000 C788607% 000003 000000 0508 0000 C788602% 000003 000000 0501 0000 C788606% 000003 000000 0496 0000 C788603% $ cut_fill CUT_FILL 1.0 - adds space delimiters to fields in fixed column file spaces in alphanumeric fields (>= 8 characters) are replaced by % Input file? beefm.ped Output file? beefm.pedcut give first and last column of each field, 0 0 ends 1 8 29 36 37 44 0 0 3 fields to extract length of record = 44 written 209663 records and filled 688111 spaces $ head beefm.pedcut B1%%%%%% 99999999 99999999 B1042%%% 99999999 99999999 B1045%%% 99999999 99999999 B1048%%% 99999999 99999999 B1049%%% 99999999 99999999 B1052%%% 99999999 99999999 B1059%%% 99999999 99999999 B10609%% 99999999 99999999 B1061%%% 99999999 99999999 B10610%% 99999999 99999999 3. Replace 9999999 in pedigrees with 0 $ head beefm.pedcut B1%%%%%% 00000000 00000000 B1042%%% 00000000 00000000 B1045%%% 00000000 00000000 B1048%%% 00000000 00000000 B1049%%% 00000000 00000000 B1052%%% 00000000 00000000 B1059%%% 00000000 00000000 B10609%% 00000000 00000000 B1061%%% 00000000 00000000 B10610%% 00000000 00000000 4. Run the renumbering program. See manual for extra information. $ renummat3 available memory in Mbytes? 50 RENUMMAT3 - renumbering tool has capacity to renumber up to approximately 409090 animals and 500000 other levels ** This version supports IDs of up to 20 characters ** input data file? beefm.data output data file? beefm.data1 printout data name? beefm.out give positions of colums to be renumbered into single factor, 0 ends 1 2 0 give minimum number of observations per level for each factor, 0 if all observations retained 0 2 numbers expected, missing numbers assigned 0 give positions of columns to pass unchanged 3 4 give position of animal in data file, 0 if none 5 name of the pedigree file beefm.pedcut name of the output pedigree file? beefm.ped2 positions of animal, sire, dam and year of birth in the pedigree file, year of birth = 0 if missing 1 2 3 0 Is it a repeatability model? (1=yes/0=no) 0 Is it a maternal model? (1=yes/0=no) 1 data files read to 200 column only Try to remove redundant animals (1=yes/0=no) 1 Remove unknown parent groups from pedigree? (1=yes/0=no) 1 Example parameter file for program JAA was stored in file renum.jaa Example parameter file for program blupf90 was stored in file renum.b90 Both files must be edited before use Summaries and messages were stored in file renum.msg 4a. Printouts of renumbering file $ cat renum.msg input data file: beefm.data output data file: beefm.data1 input pedigree file: beefm.pedcut output pedigree file: beefm.ped2 data records: 128322 original levels: 26631 levels after cutting: 26631 Renumbered factors factor levels 1 20893 2 5738 Animal factor summary pedigree records: 209663 animals found: 209663 animals eliminated: 12989 repeated pedigrees: 0 Consecutive number assignment animals with records 1 - 128322 parents with pedigrees or ties 128323 - 196674 parents for maternal effect 196675 - 197686 unknown parent groups 197687 - 197687 Example parameter file for program JAA was stored in file renum.jaa Example parameter file for program blupf90 was stored in file renum.b90 Both files must be edited before use 4b. Printout of renumbered data files. Fields are: cg1, cg2, direct, WWT, gain, maternal $ head beefm.data1 1 1 1 577 0 143202 1 1 2 645 0 165151 1 1 3 725 0 82416 1 1 4 817 0 154942 2 1 5 625 0 151187 2 1 6 642 0 164603 3 1 7 519 0 152783 3 1 8 508 0 157194 3 1 9 501 0 53122 3 1 10 496 0 57324 4c. Pedigree file. Fields are: animal, parent1, parent2, parents' code, year of birth (incorrect here), number of known parents, number of records, number of progenies, original ID $ head beefm.ped2 166336 0 0 3 2431697 0 0 1 B1%%%%%% 173192 0 0 3 2430819 0 0 2 B1045%%% 165750 0 0 3 2430819 0 0 2 B1052%%% 173193 0 0 3 2430819 0 0 3 B1059%%% 173194 0 0 3 2430819 0 0 2 B1061%%% 173195 0 0 3 2430819 0 0 2 B10612%% 165752 0 0 3 2431697 0 0 1 B1066%%% 165755 0 0 3 2431697 0 0 1 B1067%%% 173196 0 0 3 2430819 0 0 2 B1097%%% 165751 0 0 3 2430819 0 0 2 B1101%%% 4d. Single-trait Parameter file fo BLUP and accuracy programs, as created by RENUMMAT3 $ cat renum.b90 # Generated by RENUMMAT; must be edited DATAFILE beefm.data1 NUMBER_OF_TRAITS 1 NUMBER_OF_EFFECTS 4 OBSERVATION(S) 4 WEIGHT(S) EFFECTS: POSITIONS_IN_DATAFILE NUMBER_OF_LEVELS TYPE_OF_EFFECT [EFFECT NESTED] 1 20893 cross 2 5738 cross 3 197687 cross 6 197687 cross RANDOM_RESIDUAL VALUES 1.0 RANDOM_GROUP 3 4 RANDOM_TYPE add_animal FILE beefm.ped2 (CO)VARIANCES 1.0 1.0 1.0 1.0 5. Edit the paremeter file, changing the number of traits, collapsing cg effects, adding PE effect, and adding variance components. If a template exists, only a few numbers need to be changed. See details in documentation on BLUPF90. $ cat renum.b90 # Generated by RENUMMAT; already edited DATAFILE beefm.data1 NUMBER_OF_TRAITS 2 NUMBER_OF_EFFECTS 4 OBSERVATION(S) 4 5 WEIGHT(S) EFFECTS: POSITIONS_IN_DATAFILE NUMBER_OF_LEVELS TYPE_OF_EFFECT [EFFECT NESTED] 1 2 20893 cross 3 3 197687 cross 6 0 197687 cross 6 0 197687 cross RANDOM_RESIDUAL VALUES 2950 0 0 1900 RANDOM_GROUP 2 3 RANDOM_TYPE add_animal FILE beefm.ped2 (CO)VARIANCES 802 -237 266 0 -237 723 0 0 266 0 858 0 0 0 0 0 RANDOM_GROUP 4 RANDOM_TYPE diagonal FILE (CO)VARIANCES 126 0 0 0 6. Run BLUP program $ blup90iod name of parameter file?renum.b90 BLUP90IOD 2.13 Parameter file: renum.b90 Data file: beefm.data1 Number of Traits 2 Number of Effects 4 Position of Observations 4 5 Position of Weight (1) 0 Value of Missing Trait/Observation 0 EFFECTS # type position (2) levels [positions for nested] 1 cross-classified 1 2 20893 2 cross-classified 3 3 197687 3 cross-classified 6 0 197687 4 cross-classified 6 0 197687 Residual (co)variance Matrix 2950.0 0.00000 0.00000 1900.0 correlated random effects 2 3 Type of Random Effect: additive animal Pedigree File: beefm.ped2 trait effect (CO)VARIANCES 1 2 802.00 -237.00 266.00 0.00000 2 2 -237.00 723.00 0.00000 0.00000 1 3 266.00 0.00000 858.00 0.00000 2 3 0.00000 0.00000 0.00000 0.00000 Random Effect(s) 4 Type of Random Effect: diagonal trait effect (CO)VARIANCES 1 4 126.00 0.00000 2 4 0.00000 0.00000 REMARKS (1) Weight position 0 means no weights utilized (2) Effect positions of 0 for some effects and traits means that such effects are missing for specified traits Data record length = 6 G 802.00 -237.00 266.00 0.00000 -237.00 723.00 0.00000 0.00000 266.00 0.00000 858.00 0.00000 0.00000 0.00000 0.00000 0.00000 G inverse 0.15580E-02 0.51072E-03-0.48302E-03 0.00000 0.51072E-03 0.15505E-02-0.15833E-03 0.00000 -0.48302E-03-0.15833E-03 0.13152E-02 0.00000 0.00000 0.00000 0.00000 0.00000 G 126.00 0.00000 0.00000 0.00000 G inverse 0.79365E-02 0.00000 0.00000 0.00000 datafile : beefm.data1 read 128322 records round = 1 eps = 12. round = 2 eps = 3.3 round = 3 eps = 0.47 round = 4 eps = 0.15 round = 5 eps = 0.33E-01 round = 6 eps = 0.25E-02 round = 7 eps = 0.13E-01 round = 8 eps = 0.14E-01 round = 9 eps = 0.51E-02 round = 10 eps = 0.11E-02 round = 11 eps = 0.13E-02 round = 12 eps = 0.91E-03 round = 13 eps = 0.39E-03 round = 14 eps = 0.50E-03 round = 15 eps = 0.89E-03 round = 16 eps = 0.71E-03 round = 17 eps = 0.28E-03 round = 144 eps = 0.18E-09 round = 145 eps = 0.17E-09 round = 146 eps = 0.16E-09 round = 147 eps = 0.15E-09 round = 148 eps = 0.17E-09 round = 149 eps = 0.13E-09 round = 150 eps = 0.99E-10 solutions stored in file: "solutions" 6a. Solutions file $ head solutions trait/effect level solution 1 1 1 640.7411 2 1 1 0.0000 1 1 2 577.0890 2 1 2 463.8596 1 1 3 447.1501 2 1 3 441.1956 1 1 4 498.6952 2 1 4 258.3874 1 1 5 521.7836 7. Run accuracy program $ accf90 name of parameter file?renum.b90 ACCF90 1.00 Parameter file: renum.b90 Data file: beefm.data1 Number of Traits 2 Number of Effects 4 Position of Observations 4 5 Position of Weight (1) 0 Value of Missing Trait/Observation 0 EFFECTS # type position (2) levels [positions for nested] 1 cross-classified 1 2 20893 2 cross-classified 3 3 197687 3 cross-classified 6 0 197687 4 cross-classified 6 0 197687 Residual (co)variance Matrix 2950.0 0.00000 0.00000 1900.0 correlated random effects 2 3 Type of Random Effect: additive animal with unknown parent groups Pedigree File: beefm.ped2 trait effect (CO)VARIANCES 1 2 802.00 266.00 -237.00 0.00000 2 2 266.00 723.00 0.00000 0.00000 1 3 -237.00 0.00000 858.00 0.00000 2 3 0.00000 0.00000 0.00000 0.00000 Random Effect(s) 4 Type of Random Effect: diagonal trait effect (CO)VARIANCES 1 4 126.00 0.00000 2 4 0.00000 0.00000 REMARKS (1) Weight position 0 means no weights utilized (2) Effect positions of 0 for some effects and traits means that such effects are missing for specified traits Data record length = 6 Assumed management as effect number 1 Assumed animal effect as number: 2 Assumed maternal effect as number: 3 Assumed PE effect as number: 4 var ratios,direct 4.90524 2.62794 var ratios,mat 4.37296 0.000000 var ratios,pe 29.7778 0.000000 read 128322 records in 2.82000 s, Read 197686 animal pedigrees round 1 , convergence = 0.584861 round 2 , convergence = 0.773062 round 3 , convergence = 0.263571 round 4 , convergence = 0.194576 round 5 , convergence = 9.940614E-02 round 6 , convergence = 7.802030E-02 round 7 , convergence = 4.532169E-02 round 8 , convergence = 3.520789E-02 round 9 , convergence = 2.230356E-02 round 10 , convergence = 1.698911E-02 round 11 , convergence = 1.138279E-02 round 12 , convergence = 8.555759E-03 round 13 , convergence = 5.946918E-03 round 14 , convergence = 4.437561E-03 round 15 , convergence = 3.163730E-03 round 16 , convergence = 2.354941E-03 round 17 , convergence = 1.709793E-03 round 18 , convergence = 1.276756E-03 round 19 , convergence = 9.369402E-04 round 20 , convergence = 7.068740E-04 Accuracy computed as: 1-(pev/var)** 0.500000 Read file solutions solutions stored in file: "sol_and_acc", 282.550 s 7b. Output file contains combined BLUP and accuracy values $ head sol_and_acc trait/effect level solution acc mat_sol mat_acc 1 2 1 3.6021 0.163 26.0849 0.119 2 2 1 10.3552 0.074 0.0000 0.000 1 2 2 14.0040 0.105 21.7372 0.060 2 2 2 11.6611 0.027 0.0000 0.000 1 2 3 31.5352 0.130 29.4291 0.071 2 2 3 1.9160 0.037 0.0000 0.000 1 2 4 43.3728 0.111 39.3531 0.079 2 2 4 1.9272 0.030 0.0000 0.000 1 2 5 19.5176 0.092 27.4459 0.067 7c. List of all animals with direct accuracy > .8 $ awk '$5 >.8' sol_and_acc trait/effect level solution acc mat_sol mat_acc 1 2 11452 47.9349 0.827 16.5571 0.751 1 2 14973 31.1588 0.806 44.3220 0.737 1 2 19166 45.8266 0.810 47.9281 0.823 1 2 37558 55.6543 0.844 51.2356 0.784 2 2 37558 31.2728 0.867 0.0000 0.000 2 2 96158 13.0065 0.811 0.0000 0.000 1 2 174733 47.7819 0.892 28.7337 0.888 2 2 174733 16.7122 0.844 0.0000 0.000 1 2 175271 11.1233 0.831 37.8106 0.850 1 2 179377 39.5539 0.831 30.6742 0.807 1 2 181548 24.2466 0.899 41.6338 0.887 2 2 181548 9.0120 0.843 0.0000 0.000 1 2 182301 20.9249 0.814 35.6619 0.730 1 2 182388 51.7405 0.823 31.8299 0.762 1 2 184091 43.1845 0.840 2.9996 0.755 1 2 189195 10.0591 0.875 27.3229 0.885 2 2 189195 12.5291 0.821 0.0000 0.000 1 2 193764 27.0292 0.887 35.5745 0.893 2 2 193764 6.1566 0.837 0.0000 0.000 7d. List of all animals with maternal accuracy > .8 $ awk '$7 >.8' sol_and_acc 1 2 19166 45.8266 0.810 47.9281 0.823 1 2 173740 32.0919 0.799 23.2956 0.808 1 2 174733 47.7819 0.892 28.7337 0.888 1 2 175271 11.1233 0.831 37.8106 0.850 1 2 179377 39.5539 0.831 30.6742 0.807 1 2 181548 24.2466 0.899 41.6338 0.887 1 2 189195 10.0591 0.875 27.3229 0.885 1 2 193764 27.0292 0.887 35.5745 0.893 7e. To obtain original IDs, merge the solutions file (column 3) with pedigree file (column 1), adding column 9 from the last file $ head beefm.ped2 166336 197687 197687 3 2431697 0 0 1 B1%%%%%% 173192 197687 197687 3 2430819 0 0 2 B1045%%% 165750 197687 197687 3 2430819 0 0 2 B1052%%% 173193 197687 197687 3 2430819 0 0 3 B1059%%% 173194 197687 197687 3 2430819 0 0 2 B1061%%% 173195 197687 197687 3 2430819 0 0 2 B10612%% 165752 197687 197687 3 2431697 0 0 1 B1066%%% 165755 197687 197687 3 2431697 0 0 1 B1067%%% 173196 197687 197687 3 2430819 0 0 2 B1097%%% 165751 197687 197687 3 2430819 0 0 2 B1101%%% 8. All these computations can be streamlined into one batch program. Will do as experience is gained. Also, formats can be changed, and IDs may be matched in the accuracy program, without the need to do the manual matching.