Other Data Glitches to Watch For

  Even after we provide the quality flags for the RPA and IDM data there are still problems and glitches which will show up in any data set this complex and this large. On this page we list some of the other problems you may encounter and some "work-arounds" to use with them.

Jumps in the total plasma density

  There are two means of determining the total plasma density (Ni) with SSIES. The first is from the analysis of the RPA data and the second is from the scintillation meter. In all our analysis we have determined that the results from the two different instruments are identical and there is no systematic difference between them. However, the resolution of the RPA is one measurement per sweep (i.e.--one measurement every four seconds) while the scintillation meter samples the density 24 times a second. Because there are some periods when the RPA data are bad or unavailable, we chose to present the scintillation meter Ni here on the plots and in the datafiles (though at the lower resolution of one measurement every four seconds) to provide the most complete coverage. However the scintillation data has its own set of glitches.

  The scintillation meter can sample the plasma density data using five different sensitivity ranges. The information about the current sensitivity range is sent down every 16 seconds. However if the range changes during the 16 seconds between the range flag, then another range flag is sent down in the density data stream. If there is a gap in the data or if the algorithm incorrectly reads the range flag as a density measurement then there is a discontinuous jump in the plasma density output. We have worked to clean up most of these jumps, but we have not yet come up with an algorithm that will detect all of them. Part of the problem is that some routines which correct these jumps also start "correcting" the real jumps we find in the auroral region or in bubbles in the equatorial regions. So our compromise has been to leave the algorithm strong enough to get most, but not all, of the jumps rather than ruin the higher structured data. Since the remaining jumps are obvious to the eye, we leave it to the responsibility of the user to check for and correct these jumps.


   In the figure above (F13 for 16 March 2000) we can see the total plasma density as the black line in the bottom box. All the data are nominal except for the two downward jumps at the far right and expanded below.

  The first jump is downward at 81342 seconds (22:35:42 UT) where the density drops from 9.987 x10^4 ions/cc to 3.294 x 10^4, then after another 12 seconds returns to 1.13 x 10^5. Since the change here is only for about 16 seconds and the difference is almost exactly a factor of 3.162 (the square root of 10, the difference between adjacent ranges), this drop in density is almost certainly an artifact of the algorithm misreading the range change. Then there is a second drop at 81542 seconds (22:39:02 UT) where the density drops for 16 seconds from about 3.149 x 10^5 ions/cc to about 1.017 x 10^5, again roughly a factor of the square root of 10. So this too is an artifact of the algorithm misreading the range. Note that between these two drops, there is a short-lived peak in the density. While we have seen artifacts where the density jumps upwards, in this case the jump is only by a factor of 1.25, not by the square root of 10, so that increase is physically real.

  Since these jumps stand out clearly from the rest of the smoothly changing data, they are easy to spot. For now the only work-around is for the you to find them and correct the individual data points by either multiplying or dividing by the square of 10 to bring them in line with the surrounding data. Eventually we hope to develop a program that will fix these glitches reliably, but that is not currently a high priority.

Short Data Gaps

   One key assumption for the DMSP SSIES data analysis is that the data stream is unbroken with no gaps. Ideally this is what should happen, and in reality well over 99% of all the orbits do not contain any breaks. The data stream for the DMSP SSIES data is arranged like this: header data / 60 seconds of instrument data / header data / 60 seconds of instrument data / and so on. The header data gives the time and location data for the spacecraft at the beginning of that minute. Thus the location data in this dataset for each 4 second record of data is an interpolation between the locations given in the two sequential headers. But if there is a gap of one or more minutes in the data stream, then the locations given during the final 60 seconds before the gap are wrong. The analysis incorrectly stretches the locations out to fill in the data gap. Note that since the times are calculated by simply adding 4 seconds to each record (after starting at 2 seconds after the start of the minute), then the times associated with each 4-second record are correct. Below is an example of an orbit where there is a gap of several minutes in the northern polar region.

This five-minute gap occurs between 14:21 and 14:26 UT and in the plot above the gap appears as a straight line in the cross-tack velocities connecting two regions of real data. As the data in this plot are presented by time, then the location of the data and the gap are properly presented. Below is the polar dial plots from the same orbit.


Since the data here are plotted as a function of the calculated position, the gap in the northern polar region is clearly visible and the tic marks designating the flows are incorrectly spaced out to fill in the gap.

  Under normal conditions gaps like this are very rare. However, near the end of a spacecraft's lifetime these gaps become a regular feature of the northern polar pass. The DMSP spacecraft record all the data for one orbit on a tape recorder and as the spacecraft passes over the northern polar region the data are downlinked to the tracking station at Thule, Greenland. Once the downlink starts, a second tape recorder takes over the recording function while the first record plays back the previous orbit's data. One orbit later they switch roles again. There are a total of four tape recorders on each spacecraft so there are two backups. As the spacecraft ages the recorders begin to wear out until eventually there is only a single recorder that is operating. Under those conditions there is no data recorded during the 4 to 5 minute playback over Thule, so a gap exists on every orbit. F14 was down to a single recorder sometime in 1999, so all the F14 currently on line here contains these gaps as illustrated in the figures above. However, since all these data are also suspect (see question #19 in the FAQ) these gaps are not serious in the F14 data.

  Eventually this problem will appear in the F13 and F15 data and we are working to develop a fix that will correct this in the plots and the ascii data. In the meantime, should you run across a gap like this in the middle of otherwise good data, remember that the times are correct, but the location parameters (latitude, longitude, magnetic latitude, and MLT) are incorrect, but only for the final 60 seconds before the gap. The quality of the geophysical data during these 60 seconds are not affected by this.

(last updated 11 March 2003)

Back to the DMSP SSIES data main page