What is this about?
Data Integrity Checks in EKO Q help you understand whether your dataset is complete, consistent, and physically sound to set a solid base for your analyses. The checks inspect general data structure, look for irregularities on a large scale, and summarize key informative metrics.
The checks follow the requirements of industry standards such as IEC 61724-1 and ISO/TR 9901, solar industry best practices and recommendations of EKO Instruments.
Why this matters
Solar Irradiance data analysis relies on measurement data being collected correctly. Yet measurement data depends on the data acquisition system it comes from. Undetected systematic problems in data structure bias and corrupt all the analyses and lead to wrong conclusions. For example:
- Missing data during peak sun hours can underestimate performance
- Incorrect timezone can shift the entire solar profile
- Wrong units of measure require conversion before the data can be used
Industry standards like IEC 61724-1 recommend carefully selected settings and data verification before any further analysis. Data Integrity Checks look for patterns that often suggest critical issues with your data acquisition system or irradiance sensor, which in turn can compromise all the analyses that follow.
Note: Whether your data is still good for your needs depends on your application. Some checks may be relevant or not in your case. Data Integrity Checks help raise general flags to let you see your data better and decide.
How to read the results
The Data Integrity checks independently check your data for signs of typical problems, one at a time. Good data passes all of them. Faulty data may fail one or multiple checks.
If any of the tests fails, check your data and correct it.
Note, that some problems may affect multiple tests. If multiple tests fail, check your data for signs of each problem. See other sections of the report for more insights.
Data Overview
The “Carpet” plot provides a visual overview of the irradiance data during the whole period of analysis. It uses date and time of the day as coordinates and color-code irradiance values. See more in What is carpet plot.
Use this plot to visually inspect your data for irregularities. See more in How to read and understand Carper plot.
Time Zone Check
Accurate time is critical for analyses in Solar. Even a small mismatch of a few minutes can cause noticeable discrepancies with other data and lead to large errors. Time zone is basic to the time accuracy but it is often incorrectly set in filed data and leads to hours of time error. The check verifies that timestamps look aligned with the expected timezone.
Multiple checks are performed to prove or disprove that the data is using the correct time zone and suggests a correction if not.
If the test fails, check your data and provide the correct time zone before further analyses.
Sometimes the accuracy of the test is not enough to reliably detect the time zone, and an “Uncertain” result is reported. Check your data visually, using Carpet plot and other methods.
Note: Other problems in your data, such as sensor miss-orientation or shading may impact the time zone check.
Units Check
Irradiance data is expected in W/m² but may come in other irradiance or irradiation units: kW/m², Wh/m², J/m², W/ft². Mixing these can lead to incorrect analysis and misleading results while not necessarily obvious. The check verifies that the numerical values match the expected units of measure and can be trusted.
Data Availability
Data availability tests provide statistics on how well represented your data is due to record rate, data gaps and suspicious records. It represents typical recommended data quality protocols used for Solar Resource Assessment and Energy Evaluation analyses, so that you can estimate how much of your data may be discarded.
Note: Your analysis may use slightly different versions of these tests, and exact numbers may vary.
Data Availability tests report daytime and nighttime separately. Daytime values are also used to assess impact on the available solar energy metrics.
Time Granularity
This check refers to how often your data is recorded, the rate of “records” in terms of IEC 61724-1. While individual measurements often happen at a higher sample rate, only aggregated values of all the samples during the record interval must be recorded for analysis.
Record rate should be fast enough to detect all the fast changes your application requires. The faster the better but too fast records lead to large datasets, and it can become a burden. 1 minute is a universally agreed frequency for solar irradiance data in most PV applications. A 5-minute step is often still reasonable, but a slower rate may hide problems and significantly affect the quality of further analyses.
Changes in record rate during data acquisition is considered a poor practice and complicates further analysis.
Higher resolution up to 1-second provides better insights into system behavior, especially during rapidly changing conditions like passing clouds.
Lower resolution down to 1-hour may be preferred for long-term analyses. In this case, the lower resolution data for analysis should always be obtained from higher resolution measurement data.
The data should have one fixed time granularity. Switching between multiple time granularities complicates data processing and may have a significant impact on your analysis.
Note: In Starter and Standard subscription plans, EKO Q may force resampling of your data to a practical rate of 1 to 5 minutes during data onboarding. Contact EKO sales team for analysis at full rate.
Note: Timestamp granularity check in EKO Q does not reveal details of how the records are obtained, such as measurement (sample) rate, sample averaging, time-stamping convention, or post-processing of the measurement data.
Data Gaps
Gaps mean periods of time without data.
Different gaps have different impact on your data, it depends on your application. Gaps in the middle of the day typically mean more than in the middle of the night. Short gaps are often easy to fill while filing large gaps may introduce significant bias or increase analysis uncertainty.
The data gaps are reported separately for daytime and nighttime, and grouped by gap length.
Note: In this test, suspicious or erroneous records still count as records, but they may be added to the data gaps in your analysis.
Compromised Irradiance Values
These tests represent the checks commonly performed or raw irradiance data. They compare irradiance values to basic expectations based on physical models (physically impossible values etc.), and consequent records to each other (duplicates etc.). Such compromised values are often removed before analysis to lower the risks.
This section provides an estimate on how much data may be flagged erroneous. Make sure your data only has a small amount of them.
Note: Your analysis may use slightly different versions of these tests, and exact numbers may vary.
Note: EKO Q does not offer automated data correction. Suspicious records must be carefully examined separately.
