Initial Results of Quantification of Model Validation Results Using Modal Analysis

High quality generator dynamic models are critical to reliable and accurate power systems studies and planning. With the availability of PMU measurements, measurement-based approach for model validation has gained significant prominence. Currently, the model validation results are analyzed by visually comparing real–world PMU measurements with the model-based simulated data. This paper proposes metrics to quantify the generator dynamic model validation results based on the response of generators to each system mode, which includes both local and inter-area, using modal analysis approach. The metrics provide information on the inaccuracy associated with the model in terms of the characteristics of each mode. Initial results obtained using the real-world data validates the effectiveness of the proposed metrics. In this paper, modal analysis was carried out using Prony method.

Abstract-High quality generator dynamic models are critical to reliable and accurate power systems studies and planning. With the availability of PMU measurements, measurement-based approach for model validation has gained significant prominence. Currently, the model validation results are analyzed by visually comparing real-world PMU measurements with the model-based simulated data. This paper proposes metrics to quantify the generator dynamic model validation results based on the response of generators to each system mode, which includes both local and inter-area, using modal analysis approach. The metrics provide information on the inaccuracy associated with the model in terms of the characteristics of each mode. Initial results obtained using the real-world data validates the effectiveness of the proposed metrics. In this paper, modal analysis was carried out using Prony method.
Index Terms-Power plant model validation, Validation metrics, Modal analysis, Prony method, Signal Processing, PMU measurements High quality dynamic model of generators are critical to reliable and economical power system operations and planning. Dynamic studies for various system disturbances, such as generation loss, line trip, etc., is carried out using these models for both short and long term planning. These studies provide information on several aspects of power systems dynamic stability such as rotor angle stability, damping ratio of system modes, generator's contribution to the primary frequency response, system frequency and voltage recovery, etc., and identify contingencies that can result in system instability. The accuracy of these studies heavily depends on the quality of dynamic models used, thereby making validation of generator dynamic models significant. The need for accurate and upto-date dynamic models for reliable and economical grid operations and planning was reinforced after the well-known 1996 western grid blackout. The planning Western System Coordinating Council (WSCC) model could not replicate the unstable system oscillations observed following the series of events that led to the system-wide outage [1]. After this event, NERC now requires all generators having capacity of greater than 10 MVA to be validated every five years. Also, Reliability Standards MOD-026 [2] and MOD-027 [3] have been developed to provide guidelines for generator model validation.
Traditional methods for validating generator dynamic models include staged and standstill frequency response testing [4]. These methods involve physical testing of the generators This work is supported by the U.S. Department of Energy's Office of electricity through Transmission Reliability program. Pacific Northwest National Laboratory is operated by Battelle for DOE under contract DE-AC05-76RL01830. and therefore generators to be validated remain unavailable for normal operations. Even though these methods provide high quality dynamic models, these methods are technically difficult and are expensive [4]. With the availability of synchrophasor measurements, measurement-based validation methods have become widely-accepted [5]- [8]. This method requires Phasor Measurement Units (PMU) to be installed at the point of interconnection (POI) of each generator to be validated. The PMU measurements are used as play-in signals to validate dynamic model. This approach of validating dynamic models is available in several power systems simulator such as GE PSLF, SIEMENS PTI PSSE, PowerWorld Simulator and TSAT [9].
Bonneville Power Administration (BPA) and Pacific Northwest National Laboratory (PNNL) have developed Power Plant Model Validation (PPMV) tool that performs dynamic simulations using play-in PMU measurements and validates dynamic model. The detail of this tool can be found in [9]. Currently, the model validation is carried out by visually inspecting the difference between the simulated and actual measurements of real and reactive power. Similar approach is also used in the Power Plant Model Verification tool developed by ISO-New England [6]. In order to automate the process of model validation, some metrics system is required to quantify the power plant model validation results. In [9], root mean square is calculated between the actual and simulated measurements to quantify the model validation results. However, the approach is only suitable for cases when the simulated and actual measurements follow a similar shape-pattern. The frequencydomain based metrics, first defined in [10] as Magnitudeshape similarity measure, was applied for quantifying model validation results in [11]. This metric finds a weighted average of the similarity measure of the magnitude and phase spectra in frequency domain.
In this paper, a new set of metrics is proposed to quantify the model validation results using modal analysis method. The quantification metric is calculated based on the response of generators to each mode, both local and inter-area. A detailed methodology to obtain proposed metrics is included in the paper. While any modal analysis method, such as Prony [12] and Matrix-pencil [13], can be used to obtain the metrics, Prony analysis method is used in this paper to illustrate the methodology of obtaining the proposed metrics.
The rest of the paper is organized as follows: Section II briefly describes Prony analysis method. Section III presents a detailed methodology to obtain the proposed metrics for quantifying model validation results. Section IV provides results and discussion, and section V provides the conclusion.

I. DESCRIPTION OF PRONY METHOD
This section briefly describes Prony-analysis method to estimate system modes using ringdown oscillations, the detail of which can be found in [12].
1) In the first step, a discrete linear prediction model (LPM) is obtained, that fits the signal, by solving a linear leastsquares problem given by where a = a 1 a 2 · · · a n T , and n is the model order selected to obtain system mode estimates. The least squares solution of (1) is given bŷ where † denoted pseudo-inverse of a matrix. The n-th order polynomial equation is then given by 2) In the second step, mode estimates, given byλ i = ω i + jσ i for i = 1 to n, are calculated usinĝ where are the roots of the estimated n-th order polynomial equation given by (3), ∆T is the sampling time period of the measurements, ω i is the frequency of each mode in rad/sec, and σ i is the damping coefficient. The damping ratio of each i th mode is given bŷ 3) In the final step, the initial amplitude and phase of each mode, given by the phasor estimateB i , is calculated solving where The least squares solution of (6) is given bŷ Following steps 1 to 3, system mode estimates are given by (4) and mode-shape of each mode is given by (7).
B. Validation of system mode estimates Only estimating system modes is not sufficient unless the mode estimates are validated [14]. For validating mode estimates, the original signal is compared with the reconstructed signal, which is given by: The goodness of fit between the original and reconstructed signal is given by [12] Based on this goodness of fit metric, model order n can be selected to obtain the best fit between the analyzed and reconstructed signal.
C. Sorting of system mode estimates As described in [14], not all mode estimates represent actual system modes. Some of the mode estimates are spurious ones and need to be discarded. One of the ways of distinguishing actual mode estimates and spurious ones is to rank system modes based on their energy given by The system modes having insignificantly small energy as compared to the highest energy can be discarded.

II. METRICS FOR QUANTIFICATION OF MODEL VALIDATION RESULTS
This paper proposes new metrics for quantifying model validation results obtained using measurement-based approach. The metric is derived by calculating cumulative response of a generator to each system mode observed in the measurements. Two metrics are proposed in this paper, one for magnitude and the other for phase. The metric for magnitude incorporates any discrepancy associated with the damping and frequency of system modes between the model-based response and actual PMU measurements. The metric for phase calculates any phase difference between the two signal. The two metrics can either be combined as a weighted average or can be used separately. In this paper, the two metrics are used separately as this can provide information helpful for calibration, i.e., if the calibration should focus on phase shift or amplitude or both.
The metric for comparing the similarity of the magnitude of actual and simulated measurements is given by: superscript 'm' corresponds to estimates using actual PMU measurements and 's' corresponds to estimates using simulated signal, w i is the weight factor for each mode which is chosen to be the energy of each mode given by (10), p is the number of dominant modes selected out of n modes based on the energy of each mode as compared to that of the energy of the mode having the highest energy, and |.| denotes absolute value of the quantity. The metric for calculating similarity of the phase between actual and simulated measurements is given by: where | B m i − B s i | is the phase difference (in degree) of the i th mode calculated using actual and simulated measurement.
The metric obtained for each mode is weighted with its energy to obtain one single metric. This is done such that the mode having highest energy will have the highest effect on the metrics and vice-versa.
The step-wise methodology to obtain the proposed metrics is as below: 1) Pre-process PMU and simulated measurements for modal analysis using approach described in [15] that includes filtering, downsampling, etc. 2) Obtain mode estimates and mode-shapes for both preprocessed PMU and model-based measurements using (4) and (7). In this step, selection of model order is carried out for both the signals by comparing pre-processed original and reconstructed signal. Also, dominant modes are distinguished from the spurious ones by calculating energy of mode estimates using (10). 3) Calculate the two metrics that quantifies the similarity of the damping ratio, frequency, initial amplitude, and phase of each estimated dominant mode between the actual and simulated measurements using (11) and (14).

III. RESULTS AND DISCUSSIONS
Results were obtained using real-world PMU measurements recorded in the western grid to illustrate the effectiveness of the proposed metrics. Fig. 1 shows the PMU signal measured at the POI of the generator, and the model-based response of the generator obtained using PPMV tool. Before performing modal analysis, the signals were preprocessed as decribed in [15]. The sampling frequency of the measurements were down-sampled from 30 to 5 samples/sec and any outliers or missing data were removed and/or interpolated.

Estimation and validation of system modes
The model order selection is very critical to the proposed method as it can significantly affect the metrics for quantifying the validation results. For PMU measurements, model order n = 12, and for model-based simulated data, model order of n = 10 were chosen that gave the best fit between the original and reconstructed data as shown in Fig. 2. Table I and II give the mode estimates for the PMU and model-based simulated measurements. For metric calculations, mode estimates having energy less than (5%) of the highest energy were not considered.

Calculation of metrics
Using (11) and (14), metrics for measuring the similarity between the PMU measurements and model-based simulated data were calculated, which are given in Table III. If any mode observed in the PMU measurement was not observed in the mode estimated using the simulated data, the maximum error of 1 was assigned to that mode.
Based on the metrics calculated, it can be said that the dynamic model does not accurately represent the model that generated the PMU measurements and requires calibration. This conclusion can also be verified by comparing the contribution of each mode estimate to the PMU measurements and simulated generator response as shown in Fig. 3. As seen in these figures, the contribution of each mode to the PMU measurements and generator response do not have a good match.

Metrics calculated for the calibrated model
The examples used in this paper to calculate metrics for quantifying model validation results were also used in [9], where a new methodology to calibrate dynamic model was proposed. The results shown in the previous section corresponds to the model before calibration. The metrics were also calculated for the model calibrated using the methodology described in [9]. The model validation results for  Table IV gives the mode estimates obtained for the simulated data generated using the calibrated model, and Table V gives the metrics calculated. The calculated metrics indicate that calibrated model accurately represents the PMU measurements. This can again be verified based on the good match obtained between the contribution of each mode to the PMU measurements and the calibrated-model based simulated data as shown in Fig. 5.

IV. CONCLUSION
This paper proposes metrics for quantifying model validation results based on the response of generators to each system mode. These metrics can help identify generator models that do not have accurate dynamic response, and thereby help obtain models that can collectively represent accurate system oscillatory behavior and dynamic response. This method is applicable to ringdown-oscillation type of system response in which modes are observable in the PMU measurements. The initial results obtained using real-world data validates the effectiveness of the proposed metrics. However, extensive testing needs to be done to identify thresholds that can help distinguish a good model from the ones that need calibration. Once thresholds are identified, the proposed metrics will be used to automate the process of model validation. Future work will focus on this aspect of automating the model validation process and incorporating the methodology in the PPMV tool, and developing a calibration methodology based on the error metrics proposed in this paper.