HLM FAQs & Troubleshooting

HLM is capable of handling large data sets. The program has no limit on the size of the data files to be used; the only limit on this is that imposed by the system on which the analysis is done. Note that, in the case of extremely large data sets, both the creation of the SSM file and the iterative procedure may be somewhat slower than for smaller models. Do not assume that the program is "hanging" because nothing is happening- the manipulation of large data sets is always a time-consuming process for any statistical application.

Handling of missing data in HLM

HLM can handle missing data at level-1 of the hierarchy. Both pairwise and listwise deletion options are available Any level-2 or level-3 units with missing data should be removed or indicated as such in the data file prior to analysis, or use should be made of data imputation, as the program will either delete these observations or treat missing data codes as valid data values. A message concerning the presence/removal of missing data will also be printed to the *.sts file, which may be accessed using the Check Stats button in the Make SSM/MDM dialog box.

Note that, if blanks or missing data codes are present in the data files, these codes will be treated as valid data if not removed or identified as missing data prior to analysis. For more information on this, see missing data or troubleshooting.

HLM has stopped working

It may occasionally happens that WHLM/HLM2/HLM3 will stop working. Creating an SSM file using a previously used (and tested) rsp file may, for example, become impossible. This is often due to the presence of a large number of temporary files. These files, with the file extension tmp, are usually to be found in the windows\temp subdirectory of the system. Check the settings on the PC to find out what the TEMP setting is. Generation of scratch files by HLM becomes impossible when this directory is too full, causing HLM to stop running. To solve this, erase all the *.tmp files in the relevant directory.

Difficulty selecting variables

When a large SSM file is used to construct a model, the last variables on the selection list at the left of the WHLM screen may be difficult to select. Typically, the Windows taskbar appears on top of the last variables. To solve this problem, right-click the taskbar and select properties. Then click Always on top to disable this setting and return to the Windows desktop. If you then maximize the WHLM screen and scroll down the list of variables, the last variables on the list should be visible and the taskbar will simply temporarily disappear from view.

Problems creating SSM files

Most of the problems experienced when using HLM are with the creation of the Sufficient Statistics Matrix File (SSM file). In general, these problems are caused by invalid data or incorrect specifications when creating the SSM file. The most common of these problems, losing units of analysis when creating the SSM file, is discussed below.

Data in the data files are not sorted correctly. Data in the level-1 data file, for instance, has to be sorted by the ID field as used in the level-2 data file. If the data is not sorted, an incorrect number of units will be included in the SSM file. When creating a level-3 SSM file, level-2 IDs need to be sorted inside of the level-3 IDs. For example, suppose that you have only 3 observations (level-1) for 2 level-2 units. The correct order of these observations in the level-1 data file is:

1 (First observation for first level-2 unit)

1 (Second observation for first level-2 unit)

1 (Third observation for first level-2 unit)

2 (First observation for second level-2 unit)

2 (Second observation for second level-2 unit)

2 (Third observation for second level-2 unit)

If, however, you accidentally specified it as

1
2
2
2
1
1 ,

WHLM will not create the SSM file but warn you that the data in your level-1 file is incorrectly sorted. But, if you had this in your level-1 data file as the ID variable:

1
1
2
2
3
3

WHLM will create an SSM file, with only 4 level-1 units and 2 level-2 units. The observations for which no corresponding ID could be found in the level-2 data file will be ignored.

The number of units used in the creation of the SSM file will also differ from those in the data files if you have missing data. Missing data are only allowed on level-1 of the hierarchy, and you need to provide information on the missing data codes when creating the SSM file. Otherwise, these blanks and other codes, for example -99, will be treated as data. No missing data are allowed in the level-2 data file. HLM will do listwise deletion and print a warning message concerning this to the *.sts file which may be accessed using the Check Stats button from the Make SSM/MDM dialog box.
The IDs may be incorrectly specified. See the notes on the construction of an SSM file for more information on this possible problem.
If you have a very large number of level-1 units, for example, 48000, the number of level-1 units reported when the statistics for the SSM file is viewed in WHLM is currently incorrect. A user with a large data set should, until this problem is fixed, use the *.sts file or check the information given at the top of the output obtained from the HLM analysis.

Unable to import data into HLM

The most likely cause is that the DBMSCOPY utility was not installed during the installation process. This third-party piece software (DBMSCOPY, from Conceptual Software) is used to read most other binary data formats such as STATA or EXCEL as input files for HLM. If you do not have DBMSCOPY installed, re-install the program and make sure that you select DBMSCOPY to be installed. Also see the DBMSCOPY information given elsewhere.

Unable to access a created SSM file

The SSM screen in WHLM has three buttons: Make SSM file, Check stats and OK. The Make SSM file button should be clicked first in order to create the SSM file. Only after the *.sts file has been viewed using the Check stats button, should the OK button be used to exit this screen as it may be that, due to errors in the SSM file creation process and/or in the data files, the SSM file was not created. Also check that the created SSM file is larger than 0 KB in size.

Unable to find the design weights for use during analysis

If design weights are to be used during analysis, these weight variables will not be displayed in the WHLM screen. Rather, the variables to be used as design weights must be specified as such during the creation of the SSM file. In the Make SSM/MDM dialog box, the user is given the option to include design weights at level-1 and/or level-2 of the model. Information on the weight variables is then included in the SSM file and HLM will automatically use the weights assigned during the SSM file creation process. Note that design weights may only be used at levels 1 and 2. It is currently unavailable at level-3 of the model.

Non-linear analysis terminates prematurely

In the case of the non-linear models, it is important that both the SSM file and the level-1 data file used to create the SSM file remain in exactly the same directory as when the SSM file was originally created. This is because WHLM needs access to both the SSM file and the level-1 data file during the iterative procedure. In the case of linear models, only the SSM file is accessed and the fixed position of the level-1 data file as described above is not necessary.

The descriptive statistics produced by HLM are incorrect

The descriptive statistics for data read by HLM are both printed to screen and to the default statistics file hlm2ssm.sts (or hm3ssm.sts in the case of a level-3 model).

When these statistics do not correspond at all with those obtained during exploratory data analysis, it is caused by one of the following two problems:

Whether ASCII or statistical package data are used, this message is caused by the incorrect specification of missing data. Go back to the original data file in the statistical package used to construct it, and check the missing data specification for each predictor. Make sure that the codes assigned to each variable correspond exactly to the actual missing data codes used for that particular variable.
In the case of using ASCII data, the format specification may also be incorrect, causing HLM to read variables from the wrong columns in the data file.

Group X has inadequate data

This message appears after the Make SSM button is clicked. It is listed in the DOS box that appears on the screen during the SSM file creation process.

The groups listed in the DOS box have insufficient data, and are not included in the SSM file. This is usually a function of the type of deletion specified in the Make SSM screen, or, more directly, of the inclusion of a large number of variables in the data file with a large number of missing data values.

Check the structure of the particular level-2 units listed and the distributions of the level-1 variables. If there are no valid data points for a level-2 or level-3 unit when doing simple frequencies with a listwise deletion option in a statistical package, that unit will be one of those deleted when the SSM file is created. It can be remedied by removing the variable(s) with the most missing data, or else imputing the missing values for these variables.

A list of such units is also printed to the *.sts file, which may be accessed using the Check Stats button in the Make SSM/MDM dialog box.

Group X not in level-2 file

A warning message is written to the DOS box when the values of the ID variables used in the data files for the different levels of the hierarchy do not correspond. It identifies the value for which a match could not be found in the ID variable field in the other data file(s). The absence of a unit will also be apparent from the descriptive statistics in either the DOS box or the default *.sts file, where the number of units used will be less than anticipated. A list of such units is also printed to the *.sts file, which may be accessed using the Check Stats button in the Make SSM/MDM dialog box.

Frequency tables of the ID codes used in the different data files can be used to verify that all codes are present in all files.

SSM file not of recognized type

The appearance of this message is usually an indication that there are problems with the SSM file created for use in analysis. If, for example, missing data were incorrectly specified, a SSM file may be created. It may, however, be an empty file that contains no information that can be used by HLM in model building.

The best way to solve this problem is to remake the SSM file and to pay close attention to the message printed to the DOS box during SSM file creation. Use the Check Stats button in the Make SSM/MDM dialog box to access the descriptive statistics for the data and verify that these statistics correspond to the results of prior exploratory data analysis.

Any error message concerning missing data has to be attended to before model building is attempted.

OLS level-1 coefficients were computed for only x of y units that had sufficient data for estimation

This message appears in the output file in the section where the OLS coefficients and starting values are given. It may also appear at the tables containing reliabilities, and at the tables with the final fixed and random effects. In all cases it is relevant only for the section of the output where the message is printed.

Note that when it appears at the final tables, it affects the chi-square values printed in these tables and generalization on the basis of these chi-squares are inadvisable, especially if a sizable proportion of the units is excluded during the chi-square calculations.

The usual cause of this message is a lack of variability within higher level units. For example, consider level-2 model with schools as the level-2 units and students as the level-1 units. If, for a given school, all students are of the same gender this may cause the problem. Also see problem in fixed part��

Note: x of y units failed the conditioning check for inversion of the level-1 predictor matrix...

In order to perform iterations, the level-1 predictor matrix needs to be inverted. This message indicates that for a number of units a unique inverse for this matrix could not be found.

Inversion is dependent on the design matrix being of full rank. Columns of data (i.e. predictors included in the analysis) must be independent. In the example below, an example is given for a level-2 unit with 5 level-1 units nested within the level-2 unit. The first column represents the intercept term, which is by default included in any HLM model. The second column represents the scores of the 5 respondents from this level-2 unit. As the scores of all 5 respondents are very similar, the second column is almost a multiple of the first.
Intercept Score1          201          201          201          201          21
Groups that passed the various checks on X'X (X = level-1 data), invertibility, positive determinant, condition number < 1E6, will have the OLS coefficients printed out. Groups that fail either of the first two checks will not appear, and will be counted as insufficient data. Groups that fail the condition check will be counted, and produce the message noted above. The level-1 coefficients for all "acceptable" units can be requested from the Basic Specifications dialog box by setting that field to the number of level-2 groups (or some large number). The results for "acceptable" groups will be printed.

This is often caused by not enough data for certain groups. For example, let�셲 say we have a model with 3 random effects. Groups with less than 3 level-1�셲 per level-2 (using HLM2 as an example) will cause this error, and probably at least some of the groups with three records will as well.

Possible solutions to this problem are:

Scaling of the problem variable(s) may be considered, for example centering of predictors. Centering predictors reduce correlations between random effects.
If retaining a variable that is a multiple of the intercept term is a problem, the intercept term may be deleted from the model.
A mean value for a variable can be calculated at group level. The mean can then be used as a level-2 covariate instead.

There is a problem in the fixed part of the model. A near singularity is likely. Possible sources are a collinearity or multicollinearity among the predictors. We suggest that you examine a correlation matrix among the fixed effect predictors

This error message indicates that there are some of the fixed effects that have essentially the same relationship with the outcome variable. Consider the case of 10 students from the same school, with school serving as level-2 unit. Gender is to be used as a fixed effect, along with the intercept effect that is automatically included in any HLM model. In this example, the values in the second column (gender) is, with one exception, identical to the first (intercept).
intercept gender1           1  1           11           11           11           11           11           11           11           11           0
This problem can be resolved in the following ways:

If retaining a variable that is a multiple of the intercept term is a problem, the intercept term may be deleted from the model.
Using the correlation matrix to find the pair or pairs of variables responsible for this problem. If, for example, a correlation close to 0 is observed for the predictors representing age and income, only of the two predictors should be used in the model. Alternatively, a transformation of income could be considered in order to keep both variables in the model.

HLM is unable to estimate covariance components for the model specified. It is likely that either

1. one or more of the variance components is very close to zero and the reliability of the associated random effect is also close to zero, or

2. there is a collinearity or multicollinearity among the random effects. In this case, the estimated correlations among the random effects would be close to 1.0, or

3. one or more of the OLS level-1 regressions produced extreme values. Review the OLS estimates for all groups. If this is the source of the problem, use the fixtau=5 option to manually reset the tau(0) matrix.

To check option (1), the tau-matrix printed in the output file must be examined. Small values on the diagonal of this matrix indicate the variable causing this problem. The information needed to check option (2) follows directly after the tau-matrix, where the tau-matrix is given in the form of a correlation matrix.

The model should be respecified. One (or more) of the random effects must be either deleted from the model or treated as fixed.

This happens when the iteration process forms an unusable (not positive-definite) tau where usually one of the diagonal elements has gone to effectively 0, or one or more of the correlations of tau falls outside ��1 <= x <= 1. (I think that in practice, the latter is more likely.) If this happens, the output should give one some indication which variables are causing the trouble, and one of them should be fixed, or removed from the analysis entirely.

Differences between linear and non-linear models

There are a few important differences between linear and non-linear analyses in HLM:

Design weights can only be used in linear analyses. This feature has not yet been incorporated in the non-linear analyses.
In the case of a non-linear model, the level-1 data file (and preferably the SSM file, too) should be left in the same folder it was in at the time the SSM file was made. Non-linear runs access both the SSM file and the level-1 file as both micro and macro iterations are performed. If the data file has been moved, the program will not complete the analysis.

Checking for convergence of the iterative procedure

Always check the output for the iterative procedure to ensure that convergence has been reached and a stable solution has been found. If the process has not converged, this will be stated in the output file just prior to the final results. The program will also prompt the user to continue the process if the maximum number of iterations specified is reached and convergence has not yet been attained.

If the iterative procedure is simply converging slowly, increasing the number of iterations will take care of this problem. The convergence criterion may also be relaxed to make convergence easier to obtain, although this may be at the cost of reducing the accuracy of the estimates.

If an increase in the number of iterations still does not lead to convergence, or the addition of one or more predictors leads to a sudden large increase in the number of iterations required for convergence, check the elements of the Tau matrices at all levels of the hierarchy. Small diagonal elements in these matrices indicate that negative variances may have been found, and that attempts by the program to fix this (also see Basic Specification menu for options) may have been unsuccessful.

Intervention by the program in such cases is due to the use of the EM algorithm for estimation in HLM2/HLM3, and the unrestricted sections of HMLM/HMLM2. In the case of special models fitted using HMLM/HMLM2, (homogenous etc.), the Fisher accelerator is used and, if a negative variance is encountered, the program will exit with a message concerning the problem encountered during estimation.

The message "This program cannot be run in DOS mode" shows up when you try to run one of the HLM 5 modules from a "DOS box" under Windows 95 or 98

The HLM 5 modules (HLM2.EXE, HLM3.EXE, HMLM.EXE, and HMLM2.EXE) are so-called Win console programs. They cannot be run in DOS mode.

The DOS box under Windows NT runs Win console programs only. The DOS box under Windows 95 and 98 runs Win console programs as long as the checkbox Prevent MS-DOS-based programs from detecting Windows is not checked. This is the default situation and users will normally not run into this problem.

However, if you try to run one of the HLM 5 modules from a DOS box under Windows 95 or 98 and the message This program cannot be run in DOS mode shows up, you should uncheck this checkbox.

Right click on the DOS Prompt shortcut on the desktop or right click on the title bar of a Windowed DOS box and click on Properties.

Select the Program tab and click the Advanced button.

Uncheck the checkbox Prevent MS-DOS-based programs from detecting Windows on the Advanced Program Settings dialog box and click OK to return to the MS-DOS Prompt Properties dialog box and click OK again to implement the change.