Test Procedure Working Groups » IEC 61850 TPWG » Test Procedure Issues » Server

Rev 1.1 of the TP for Server have reduce on purpose the tests for the subscribers. There is no need to test all variant. We agreed and published with A4.11b to focuss on a randmo supported configuration with maximum number of currents (x) and voltages (y)....
There is no need now to force new tests for all configurations.
SV publisher as well in sSvp1 !use the maximum SV configuration".
No additional tests.

Intent is replace the "random" test with a deterministic test to ensure repeatability of testing.
No additional tests are proposed except in the rare case when the DUT has an ambiguous definition of "maximum test" in which case exactly 1 test is added.
This rare case is actually the example in 61869-9 where max is 24 voltage channels and 24 current channels but obviously both not at the same time!
In the proposal, what is meant for this case would be to make 2 tests: one with 24 voltage channels and 1 with 24 current channels.

Resolutions from 08-March-2022 TPWG meeting for SV publishers:
PIXIT Svp14 "Maximum number of simultaneous data channels in output stream" with value <N> will be added
Each combination of declared sampling rate and number ASDU will be tested with maximum number of voltage and current channels (suggest sSvp4 replacing the "random configuration")
Expected result is to verify in the samples:
- plausibility of the sample rate (inter-sample interval approximately reciprocal of sample rate)
- value of noASDU
- length of dataset payload = number of samples/8
If LPHD.NamVariant specifies a sum of channels greater than Svp14 for a specific combination sampling rate/number of ASDUs then 2 maximum channel tests will be performed:
(for example, Svp14 = "24" but NamVariant = "F4800S2I0-16U0-16”)
- Maximum number of NamVariant current channels plus remainder of available channels filled with voltages (for example: F4800S2I16U8)
- Maximum number of NamVariant voltage channels plus remainder of available channels filled with currents (for example: F4800S2I8U16)
Certificate will individually declare the tested variants for subscribers and publishers in the format of NamVariant (example:"F4000S1I4U4;F4800S1I4U4;F4800S2I18U8;F4800S2I8U16” without quotes)

For other tests specifying "random configuration", the chosen dataset will have 1 current channel and 1 voltage channel if both current/voltage are supported else just 1 channel

Is a PIXIT entry the correct place for limitation? If they are limitation in the configuration, the configuration probably should be handled by the ICT and not leave it to the SCT.
As soon as the ICT is responsible for the configuration, it make no sense to test all combination, but only the random one.

Responding to Thierry's question "is PIXIT entry the correct place for limitation? [It should instead be in ICD file]?"
No, PIXIT is the wrong place, but awaiting an update to 61850-6 will not help now for a conformance test.
In the past, UCA has added PIXIT entries while awaiting 61850-6 updates. After update, the PIXIT entry is deprecated and the ICD (or other) file content is used instead of the PIXIT.
I suggest that UCAIug continues this same pattern of "add PIXIT entry in advance of expected 61850-6 updated" then "deprecate after 61850-6 updated"

I disagree with the changes proposed by Bruce

1) the proposed modification for sSvp4 is verifying each combination for {F and S} associated to the maximum dataSet and additionally repeat the test with max current and max voltage.

The is no added value to the tests. sSvp1 is alread taking care of testing the maximum configuration, and the revision 1.1 decrease the number of combination tests on purpose, because otherwise it is unpractible for testing flexible device.
The decreasing of tests was agreed because the lab is chosing "a random and the maximum configuration".
I however agree that the certificate indicates the tested configurations.

2) the proposed modification for sSvs7 - for the same reason, I disagree with testing 2 different max configuration as soon as the x+y (Svs2b) imay be bigger than 24.
The random max configuration is chosen by the lab and can be documented on the certificate. We do not test by GOOSE subscription max subscription using SPS, max subscription using DPC, and so on.
Multplying the tests increases the certifcation fees without adding results.

The latest proposal to sSvp4 was not intended to execute 3 tests but only, at most, 2 tests (and even 2 tests only upon what TPWG considers rare conditions).

Reason for not wanting the lab to select a random "maximum test" is that 2 testers can then issue 2 different certificates and we lose a guiding principle of UCAIug testing of "equivalency of test lab". This principle is required to ensure that vendors cannot influence "what the lab tests" by private means (other than in public documents) and therefore inhibits vendors from choosing labs based upon "which has simpler==lower-quality testing". It also inhibits users from requiring that vendors test as specific test labs (because all labs are demonstrated to be equivalent).

The opinion that sSvp1 tests "everything needed in the maximum dataset size configuration" is not true. sSvp1 only verifies the "delay time" aspect of communications and nothing else.

In general, if you carefully inspect the tests, you will see that each test independently verifies a different aspect of the sample stream and in fact the same stream capture could be used for many of the steps (there is no need to repeatedly capture the same information).

"2 testers can then issue 2 different certificates and we lose a guiding principle of UCAIug testing of "equivalency of test lab". "

This is always the case as soon as the object model of the server contains multiple candidate to perfor a test: Which CSWI.Pos was used for the test sSBOes test? Q1CSWI1? Q2CSWI2? We do not mention which objects were used for a given test procedures. In the current discussion, we prpose to add the randmo tested variant on the certificate.

as sSVp1 is verifying the MaxDl prpoerty of the SMV Publisher, the testing tool is obliged to verify every single channel to determine if the MaxDl is respected.
I would therefore say the testing is verifying indirectly the entire telegram.

Please explain the concept of "testing is verifying indirectly the entire telegram" because expected results steps should be modified to perform these "indirect" tests.

as sSVp1 is verifying the MaxDl prpoerty of the SMV Publisher, the testing tool is obliged to verify every single channel to determine if the MaxDl is respected.

this means that the subscriber tool is capable of verifying the value of every individual dataSetMembers, which is actually a very deep analysis of the telegram.

Thierry's proposl is attempting to verify both the maxDl per individual channel, but testing each individual channel is not the goal of this issue. The goal of this issue is to test the maximum capabilities of the IED, and in this case it is the maximum number of CT and VT channels supported by the IED that can be used in a single SV application/control block. The SMVsc.max defines the maximum number of control blocks, and we want to test worst-case scenario, so only a single SV application/control block is required. Therefore the maximum is defined by the name variant F####S# I#U#, and this needs to be validated to ensure consistency across test labs. We need to prevent the scenario where Lab A tests the worst case scenario and fails a device (for example), and Test Lab B tests the same device using a random set of current/voltage channels and passes that same device. "Randomness testing" should not be used on tests that validate tangible capabilities of a device.

Dustin Tessier wrote in #note-13:

Thierry's proposal is attempting to verify the maxDl per individual channel, but testing each individual channel is not the goal of this issue. The goal of this issue is to test the maximum capabilities of the IED, and in this case it is the maximum number of CT and VT channels supported by the IED that can be used in a single SV application/control block. The SMVsc.max defines the maximum number of SV control blocks, and we want to test worst-case scenario, so only a single SV application/control block is required. Therefore the maximum is defined by the name variant F####S# I#U#, and this needs to be validated to ensure consistency across test labs. We need to prevent the scenario where Lab A tests the worst case scenario based on I#U# being included in single SV CB and fails a device (for example), and then Test Lab B tests the same device using a random set of current/voltage channels and passes that same device. "Randomness testing" should not be used on tests that validate tangible capabilities of an IED.

This is not my proposal, this is the test case description.
I am saying that the sSvp1 is verifying the biggest DataSet that can be used. Indicated in PIXIT as x+y.
sSvp1 is verifying maxDl using the maximum configuration (potentially random but maximum).
therefore the maximum configuration is fully tested in sSvp1.
I am saying that if max x is 24, max y is 24 and max x+y is 24, sSvp1 is verify x+y = 24 using a random configuration to achieve 24: 12+12, or 16+8, or 8+16 - we don't care. 1 random test is sufficient.

Thierry Dufaure wrote in #note-15:

This is not my proposal, this is the test case description.
I am saying that the sSvp1 is verifying the biggest DataSet that can be used. Indicated in PIXIT as x+y.
sSvp1 is verifying maxDl using the maximum configuration (potentially random but maximum).
therefore the maximum configuration is fully tested in sSvp1.
I am saying that if max x is 24, max y is 24 and max x+y is 24, sSvp1 is verify x+y = 24 using a random configuration to achieve 24: 12+12, or 16+8, or 8+16 - we don't care. 1 random test is sufficient.

Either the dataset is random or it is the maximum. The maxdl is validating the maxDl, so am not sure why you're suggesting this test validate both. The maximums (x/y) are defined by the name variant F####S#IXUY and the PIXIT, so there is a correlation between these two that needs to determine the IED's maximum capabilities. Neither of these (F####S#IXUY & PIXIT) have anything to do with maxDl, which is why we're suggesting dedicated tests. Please explain the resistance to having a dedicated test, rather than using a non-related tests that depends on random configurations?

@Dustin

Because this is how the test procedures we are arguing about, are about.!

The test lab chooses a random or the maximum configuration to perform each test case, with the following exceptions:
- sSvp1 shall be performed using the maximum configuration
- sSvp3 and sSvp5 shall be executed with a preferred configuration and also with a backwards compatible configuration
- sSvp10 and sSvp13 shall be executed with a backwards compatible configuration

sSvp1 is performed using the maximum configuration AND, sSvp1 is about
"Verify that the maximum delay time from taking the sample to sending the corresponding message is within the limit"

Using the max configuration is of relevance for testing the MaxDl, as bigger telgrams may need more time to be sent.

I am saying that A max confiuration test is sufficient, and that we do not need 2 tests : max configuration achieved using current channels, and max configuration using max voltages!

The proposal from 26. April is unfortunately not corresponding to the text:
"This will result in exactly 3 configurations for testing (2 legacy and 1 preferred).".
At least this is not the way I understand the proposed modifications.
I nevertheless quite agree that 3 configurations for testing is a good compromise and is already what rev 1.1 is adressing (max, legacy, and random from the preferred). Modification of sSvp1 to achieve the max configuration is ok in the proposal.
legacy configuration test can only be limited to I4U4, since this is how legacy dataSet was defined.

To Richard comment: there is no requirement that every preferred F&S have same dataset ranges. That is why sSvp6 uses each preferred with dataset I4U4 instead of max.
To Thierry: agree that more than 3 variants will be tested but this was actually done to maximize the list of "tested" variants that could appear on-certificate/in-database.
Variants are:
- "max preferred" in sSvp1/2/3/5/9/12/14/15
- every legacy (up to 4-5) and every (up to 3) preferred F&S with I4U4: sSvp6

Other tests repeat some of those variants:
- slowest legacy + "max preferred" with I4U4: sSvp7
- fastest legacy: sSvp8
- slowest legacy: sSvp10/11/13/16/17

To summarize, the publisher needs
- 2 physical configurations: I4U4 + "max configuration"
- maximum 5 F&S configurations for legacy (assuming F5760 is used), all using I4U4
- maximum 4 logical configurations preferred (assuming F96000 are declared and max configuration is not I4U4)

This allows up to 9 "tested configurations" to be listed on the certificate

I think this is a good balance between "cost of testing" and "ability to declare tested variants"

I agree with Bruce's latest proposal:

To summarize, the publisher needs
- 2 physical configurations: I4U4 + "max configuration"
- maximum 5 F&S configurations for legacy (assuming F5760 is used), all using I4U4
- maximum 4 logical configurations preferred (assuming F96000 are declared and max configuration is not I4U4)
i.e:
- "max preferred" in sSvp1/2/3/5/9/12/14/15
- every legacy (up to 4-5) and every (up to 3) preferred F&S with I4U4: sSvp6

I disagree with the addeed value of:
"Other tests repeat some of those variants"
-> as soon as all FS variants were tested within sSvp6, there is no need to repeat test involving changing configuration for the other tests.

"Other tests repeat some of those variants" meant that the other tests introduce no additional configuration.
But each of the tests requires "some" configuration" to be used and "one of the I4U4 configurations" seemed easier to me that "always using maximum-sized variant" would increase testing time/cost with no added benefit.

TPWG decisions from 17-May-2020 concerning upcoming Redmine certificate database changes:
A. Database entry "SV Nominal Frequencies:" will be removed
B. Database entries for each SV publisher and SV subscriber will have 3 entries:
1. List of one-or-more tested backwards compatible (legacy) variants taken from list: F400S1I4U4, F4800S1I4U4, F5760S1I4U4, F12800S8II4U4, F15360S8I4U4
2. List of one-or-more tested preferred rates taken from list: F4800S2, F14400S6, F96000S1
3. List of preferred variants tested (free-form text semicolon separated, example: "F4800S2I16U16;F14400S2I3U3")

Bruce Muschlitz wrote in #note-28:

TPWG decisions from 17-May-2020 concerning upcoming Redmine certificate database changes:
A. Database entry "SV Nominal Frequencies:" will be removed
B. Database entries for each SV publisher and SV subscriber will have 3 entries:
1. List of one-or-more tested backwards compatible (legacy) variants taken from list: F400S1I4U4, F4800S1I4U4, F5760S1I4U4, F12800S8II4U4, F15360S8I4U4
2. List of one-or-more tested preferred rates taken from list: F4800S2, F14400S6, F96000S1
3. List of preferred variants tested (free-form text semicolon separated, example: "F4800S2I16U16;F14400S2I3U3")

What happens to the fields:
1) SV Sampling Rate:
2) SV 61869-9 Variant Tested:
-> I think they also need to be removed now that we have concluded above mentioned B1,2,3

Agree with Thierry: Existing 3 fields for SV will be deleted:
SV nominal frequencies: These can be derived from legacy variants
SV Sampling Rate: These can be derived from legacy variants + preferred rates
SV 61869-9 variant tested: This is replaced with preferred variants tested

There will need to be some translation of existing 9-2LE certificates already in the database (30 certificates for 9-2 LE testing exist. Almost all declare F4000S1 and F4800S1 with nominal frequencies 50/60 respectively. The first example in the database is https://redmine.ucaiug.org/issues/1866

I reviewed and agree with the latest proposal: R5308_proposal_Ed2p1_20220426_ReviewedRichard_UpdatedByBruce_ReviewedThierry_UpdateByBruce