TrxShell [ mangaberri.com ]

Name	Size	Permission
diskmodel	[ DIR ]	drwxr-xr-x
images	[ DIR ]	drwxr-xr-x
importdata	[ DIR ]	drwxr-xr-x
pmie	[ DIR ]	drwxr-xr-x
pmview	[ DIR ]	drwxr-xr-x
portability-and-qa.html	27.78 KB	-rw-r--r--

Code Editor : portability-and-qa.html

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<html>
<head>
	<meta http-equiv="content-type" content="text/html; charset=utf-8">
	<meta http-equiv="content-style-type" content="text/css">
	<link href="pcpdoc.css" rel="stylesheet" type="text/css">
	<link href="images/pcp.ico" rel="icon" type="image/ico">
	<title>PCP Portability and QA</title>
</head>

<style>

table, th, td {
    border: 1px solid black;
    border-collapse: collapse;
    border-spacing: 2px;
    padding: 4px;
}

td {
    text-align: right;
}

code {
    color: #000060;
    font-size: 10pt;
    margin-left: 20px;
}

</style>

<h1 align="CENTER" style="margin-top: 0.48cm; margin-bottom: 0.32cm">
Portability Guidelines for PCP QA
</h1>
Ken McDonell &ltkenj@kenj.id.au&gt
September 2018

<h2>Background</h2>

In this document the generic term &quot;component&quot; will be used to describe a piece of
functionality in Performance Co-Pilot (PCP) that could be any of the following:
<ul>
<li>a library</li>
<li>a method or module or class in a library</li>
<li>an application</li>
<li>a PMDA</li>
<li>a service or API</li>
<li>an object in the file system</li>
</ul>

Obviously the functional goals and feature focus of PCP has been
to build tools that help understand hard performance problems that
are seen in some the the world's most complex systems. Over more
than two decades, the delivery of components to address the feature
requirements has been build on engineering processes and approaches
with a technical focus on:

<ul>
<li>
Portability

From the outset we were concerned with making PCP work successfully
across a wide variety of environments.

The earliest PCP architectural deliberations began at Pyramid
Technology in 1993, but by this time I had already been working on
portable software for more than 15 years. Then the PCP incubator moved
to SGI, and in the very early days of Linux, Mark Goodwin undertook
an &quot;IRIX to Linux&quot; port as a skunk works project within SGI
Engineering. A little later, SGI's Clustered File System (CXFS) had
PCP components for IRIX, Windows, HPUX, AIX, MacOS and Linux, across
a variety of CPU architectures, C compilers and operating systems.

Although many of these have passed away, PCP survives and is
now actively supported on all Linux variants (most importantly the
Fedora/RPM, OpenSuSE/RPM and Debian/dpkg platforms and their respective
derivatives), the BSD family, OpenIndiana (nee Sun Solaris),
Mac OS X and Windows.

So there is a very rich history of engineering for portability
in the PCP DNA, and an expectation that new PCP components will be
&quot;robust&quot; (see below for a definition of &quot;robust&quot;)
across all of the supported platforms and environments.

</li>
<li>Testability

The PCP project has never had a dedicated QA team, so by
necessity the QA model that was adopted (and at times enforced)
was one where if an engineer added a feature or fixed a bug, there
was an expectation that there would be additional QA coverage for the
associated changes. Although this approach suffers from tunnel-vision
and common assumptions between development and testing, it does have
the advantage that testing and QA is a communal responsibility.

This effort grew into the very large QA infrastructure that lives
below the qa directory in the source tree and is shipped in
the pcp-testsuite package. This represents a significant
engineering effort in its own right as the table below shows:

<table>
 <tr><th>&nbsp;</th><th>below the src dir</th><th>below the qa dir</th></tr>
 <tr><td>C or C++</td><td>320,000</td><td>37,000</td></tr>
 <tr><td>shell</td><td>23,000</td><td>85,000</td></tr>
 <tr><td>perl</td><td>18,000</td><td>700</td></tr>
 <tr><td>python</td><td>17,000</td><td>200</td></tr>
<caption>Lines of Code</caption>
</table>

The QA test suite comprises close to 1,100 test scripts. The scripts
are used by individual developers to test code changes and check
for regressions. In addition the entire suite of scripts is run
regularly across the 30+ machines in the Melbourne QA Farm.

</li>
</ul>

<h2>Robustness</h2>

Robustness simply means that every PCP application and service
either works correctly or detects that the environment will not support
correct execution and is either omitted from the build or omitted from
the packaging or reports a warning and exits in an orderly fashion.

Some examples may help illustrate this philosophy.
<ul>

<li>The platform PMDAs (linux, windows, freebsd, openbsd, netbsd,
opensolaris, aix, ...) are only compiled and packaged if the build
is being run for the corresponding target platform.</li>

<li>Optional headers are checked for
using the AC_CHECK_HEADERS() macro in <a
href="https://github.com/performancecopilot/pcp/blob/main/configure.ac">configure.ac</a>
and cpp conditionals and macros are used to guard
the inclusion of these headers, e.g. &lt;strings.h&gt; in <a
href="https://github.com/performancecopilot/pcp/blob/main/src/dbpmda/src/util.c">src/dbpmda/src/util.c</a>.</li>

<li>In the Apple operating systems there used to be a
rpc.server.nqnfs.leases metric but this has been deprecated and is no
longer available. The corresponding darwin PMDA uses PMAPI protocols
to return the type of this metric as PM_TYPE_NOSUPPORT
which all PCP clients are able to handle appropriately (either
ignore the metric or report an explanatory warning). See <a
href="https://github.com/performancecopilot/pcp/blob/main/src/pmdas/darwin/pmda.c">src/pmdas/darwin/pmda.c</a></li>

<li>Every PCP QA test script completes with a status of pass, fail
or notrun. The latter is used to indicate that the test either cannot
be run because some required component is missing or the test is not
appropriate for the current platform. See the tests for the Python
OrderedDict module and the pmrep application in <a
href="https://github.com/performancecopilot/pcp/blob/main/qa/035">qa/035</a>.</li>

</ul>

<h2>Source Code and Compile Time Portability</h2>

Mostly what's been done here is common and good engineering
practice. For example using configure, conditionals in the
GNUmakefiles and assorted sed/awk rewriting scripts
to ensure the code will compile cleanly on all platforms. Compiler
warnings are enabled and builds are expected to complete without
warnings. And in the source code we demand thorough error checking
and error handling on all system and library calls.

We've extended the normal concept of macros to
include a set of globally defined names that are are used
for path and directory names, search paths, application
names and application options and arguments. These are defined in <a
href="https://github.com/performancecopilot/pcp/blob/main/src/include/pcp.conf.in">src/include/pcp.conf.in</a>,
bound to values at build time by configure and installed
in /etc/pcp.conf. These can then be used in
shell scripts and applications in C, C++, Perl, Python
have run-time access to these via pmGetConfig()
or pmGetOptionalConfig(), see for example <a
href="https://github.com/performancecopilot/pcp/blob/main/src/pmie/src/pmie.c">src/pmie/src/pmie.c</a>.

Even file pathname separators (/ for the sane world, \ elsewhere)
have been abstracted out and pmPathSeparator() is used to
construct pathnames from directory components.

At a higher level we don't even try to build code if it is not
intended for the target platform.

<h2>Packaging Portability</h2>

At packaging time we use conditionals to include only those
components that can be built and are expected to work for the target
platform.

This extends to wrapping some of the prerequisites in conditionals
if the prerequisite piece may not be present or may have a different
name.

For Debian packaging this means debian/control is build from <a
href="https://github.com/performancecopilot/pcp/blob/main/debian/control.main">debian/control.main</a>
and the ?{foo} elements of the Build-Depends and
Depends clauses are conditionally expanded by the <a
href="https://github.com/performancecopilot/pcp/blob/main/debian/fixcontrol.main">debian/fixcontrol.main</a>
script during the build.

For RPM packaging this means using configure to
perform macro substitution to create build/rpm/pcp.spec from <a
href="https://github.com/performancecopilot/pcp/blob/main/build/rpm/pcp.spec.in">build/rpm/pcp.spec.in</a>
and using %if within the spec file to selectively include packages
and adjust the BuildRequires and Requires clauses.

<h2>QA Portability</h2>

There are purpose-designed QA applications in the
<a href="https://github.com/performancecopilot/pcp/blob/main/qa/src">qa/src</a>
directory
and the source code for these applications should follow all of the
same guidelines for portability and build resilience as those that
apply for the applications and libraries that ship with the main part
of PCP.
The only possible exception is that error handling can be a
little more relaxed for the QA applications as they are used in a more
controlled manner.

<h3>Use /etc/pcp.conf</h3>
All QA test scripts will have the variables set in /etc/pcp.conf
placed in the environment using $PCP_DIR/etc/pcp.env which is
called from
<a href="https://github.com/performancecopilot/pcp/blob/main/qa/common.rc">common.rc</a>
(with $PCP_DIR typically unset).
And all QA tests source common.rc, although the sequence is a little
convoluted (and irrelevant for this discussion).
The important thing is that the following environment variables
become available to improve the portability of QA test scripts
and the <a href="#filtering">output filtering</a> (see below)
they must perform.
<ul>
<li>A QA script should never use a literal pathname to a PCP
installed file or directory because there will be a $PCP_<something>
variable that can be used. For example: 
<code>$ cd $PCP_PMDAS_DIR/mypmda</code> 
is always correct, whereas 
<code>$ cd /var/lib/pcp/pmdas/mypmda</code> 
may not work on some platforms.</li>
<li>The current platform is available via $PCP_PLATFORM.
The common values are linux (of any flavour), freebsd,
netbsd, openbsd, darwin (Mac OS X), solaris
(really OpenIndiana) and mingw (Windows).
This can be used to make high-level decisions about the behavior
of a QA test, see the <a href="#notrun">Not run</a> and
<a href="#alternate.out">Alternate .out files</a> sections below.
</li>
<li>If a QA tests depends on a particular version or versions of
PCP, then $PCP_VERSION can be tested.</li>
<li>Because there may be more than one version of a (non-PCP) command
installed or because we need special command line arguments to make
a (non-PCP) command behave in a standard manner, the following are
available:
<ul>
<li>$PCP_AWK_PROG Posix-compliant awk</li>
<li>$PCP_SORT_PROG Unix-like sort</li>
<li>$PCP_MAKE_PROG the preferred gmake</li>
<li>$PCP_CC_PROG the preferred gcc</li>
<li>$PCP_ECHO_PROG the standard echo and importantly the
prefix ($PCP_ECHO_N) and the suffix ($PCP_ECHO_C) to allow output
without a trailing newline. These are important because 
<code>$ echo -n "Prompt? "</code> 
is not portable, nor is 
<code>$ echo "Prompt? \c"</code> 
but this portable and correct, albeit ugly! 
<code>$ $PCP_ECHO_PROG $PCP_ECHO_N "Prompt? $PCP_ECHO_C"</code> 
</li>
<li>$PCP_PS_PROG the preferred ps and more importantly when
$PCP_PS_PROG is run with $PCP_PS_ALL_FLAGS as the argument we
get the fields and field-order that a QA test can rely upon.</li>
<li>$PCP_PYTHON_PROG the version of python that the installed PCP
components are expecting (critical for version 2 vs version 3
differences).</li>
</ul>
</ul>

<h3>Use the predefined variables</h3>
PCP QA scripts follow a standard template when created with the
<a href="https://github.com/performancecopilot/pcp/blob/main/qa/new">qa/new</a>
script (which is the recommended way to create new QA test scripts)
and then the following local shell variables are available:
<ul>
<li>$seq: the current test sequence number</li>
<li>$tmp: a private and unique path that maybe used to create
temporary files or a temporary directory all of which will be
removed when the script ends. For example: 
<code>$ ... >$tmp.out 2>$tmp.err</code> 
<code>$ mkdir $tmp</code> 
</li>
<li>$sudo: run the sudo command with arguments that make it work
the same everywhere</li>
<li>$DSO_SUFFIX: if you need to refer to a shared
library by name, avoid the literal
".so" suffix; instead use .$DSO_SUFFIX that will become
.so or .dll or .dylib or ... and will work across all
platforms.</li>
</ul>

<h3>check-vm is your friend, not the enemy</h3>
There is a a very large set of applications and packages
outside of PCP
that are required to build PCP from source and/or run PCP QA.
The script
<a href="https://github.com/performancecopilot/pcp/blob/main/qa/admin/check-vm">qa/admin/check-vm</a>
tries to capture all of these requirements.
check-vm should be used as follows:
<ul>
<li>Before starting a new QA cycle, running this script with
a -p argument will identify any packages that you should consider
installing. Without the -p you'll see more details of what's
needed and why.</li>
<li>If you are a developer and add a new component, please make
sure that any additional dependencies for you new component have been
captured in qa/admin/check-vm.</li>
</ul>

<h3>Avoid exotic commands and non-standard command arguments</h3>

Our QA tests scripts are run with sh not bash and on
some platforms these are really different!

Things like an == operand
used with
test (aka [) as in 
<code>$ if [ "$foo" == "bar" ] ...</code> 
will not work with the Bourne shell.
Instead, use the = operator, e.g. 
<code>$ sh</code> 
<code>$ x=1</code> 
<code>$ [ "$x" = 1 ] && echo yippee</code> 
<code>yippee</code> 
<code>$ [ "$x" == 1 ] && echo boo</code> 
<code>sh: 3: [: 1: unexpected operator</code>

Also and any use of the bash [[ operator 
<code>$ ... [[ ... ]]</code> 
is going to blow up when presented to a real Bourne shell.
In most cases test (aka [) can be used for a
straight forward rewrite.

Even less portable is any use of the bash $((...))
construct for in-built arithmetic.
For these cases, you'll need equivalent logic using expr, e.g. 
instead of 
<code>x=$(($x+1))</code> 
use 
<code>x=&#96;expr $x + 1&#96;</code>

Another recurring one is the -i argument for sed;
this is not standard and not supported everywhere so just do not use
it. The alternative: 
<code>$ sed ... &lt;somefile &gt;$tmp.tmp; mv $tmp.tmp somefile</code> 
works everywhere for most of the cases in QA.
If cross-filesystem linking and a lame mv are in play then the
following is even more portable: 
<code>$ sed ... &lt;somefile &gt;$tmp.tmp; cp $tmp.tmp somefile; rm $tmp.tmp</code> 
If permissions are in play, then you may need: 
<code>$ sed ... &lt;somefile &gt;$tmp.tmp; $sudo cp $tmp.tmp somefile; rm $tmp.tmp</code>
Do not use seq as it is not portable.
For example, this is not portable: 
<code>for i in $(seq 4); do ...; done</code> 
but it can be rewritten as follows and this will work everywhere: 
<code>i=1; while [ $i -le 4 ]; do ...; i=`expr $i + 1`; done</code>

<h3><a name="filtering"></a>Output filtering</h3>
Each QA test script must produce a standard output stream that
is deterministic.
The
<a href="https://github.com/performancecopilot/pcp/blob/main/qa/check">qa/check</a>
script that is responsible for running each QA test script (say NNNN)
captures the standard output from NNNN and if this matches the expected
result in NNNN.out then then test is deemed to have passed, otherwise
the standard output from NNNN is saved in NNNN.out.bad and the
test is deemed to have failed.
So it is critical that output of NNNN is deterministic across
all platforms and timezones.
Applications used in a QA test script will potentially
produce output that is irrelevant
to the success or otherwise of the test.
The most obvious example is the current date from an application
run on the test system.
Consequently, most QA test scripts include one or more _filter() functions
that are responsible to translating the raw output from the applications
run in the test into the deterministic output.
These filtering functions in turn often use a set of standard functions in
<a href="https://github.com/performancecopilot/pcp/blob/main/qa/common.filter">qa/common.filter</a>
that may be applied to the output of common PCP commands and operations.
There are way too many filtering cases to describe them all here, but the list below
is illustrative of the range of techniques that may be needed.
And each QA test script may need employ more than one of the techniques
to produce deterministic output for &quot;correct&quot;
execution across all platforms.
<ul>
<li>
For many operations involving starting or stopping PCP daemons and for
common output formats from PCP commands there are existing filtering
functions in
<a href="https://github.com/performancecopilot/pcp/blob/main/qa/common.filter">qa/common.filter</a>
that may be used, for example:
<ul>
<li>_filter_pcp_start() filters the output from running the pcp
&quot;init&quot; script that starts pmcd and pmlogger.
May also be used for the separate pmcd and pmlogger
&quot;init&quot; scripts that start just pmcd and pmlogger
respectively.</li>
<li>_filter_pcp_stop() filters the output from running the pcp
&quot;init&quot; script that stops pmcd and pmlogger.
May also be used for the separate pmcd and pmlogger
&quot;init&quot; scripts that stop just pmcd and pmlogger
<li>_filter_pmie_start() filters the output from running the pmie
&quot;init&quot; script that starts pmie.</li>
<li>_filter_pmda_install() filters the output from installing a
new PMDA.</li>
<li>_filter_pmcd_log() filters the log file from pmcd.</li>
<li>_filter_pmlogger_log() filters the log file from pmlogger.</li>
<li>_filter_pmdumplog_log() filters the log file from pmdumplog.</li>
<li>_filter_pmdumptext_log() filters the log file from pmdumptext.</li>
<li>_filter_optional_pmdas() removes lines containing the names
of executable PMDAs that may, or may not, be installed.</li>
<li>_filter_top_pmns() removes lines containing the just the
top-level PMNS names for PMDAs that may, or may not, be installed.</li>
<li>_filter_pmie_log() filters the log file from pmie.</li>
<li>_filter_dumpresult() filters the diagnostic output from
pmDumpResult() (translate timestamp to a literal TIMESTAMP,
remove data addresses, replace metric values by literals
like NUMBER, HEXNUMBER, STRING or
AGGREGATE).</li>
<li>_filter_dbg() strips timestamps from some of the diagnostics
generated by the -D command line option to PCP commands.</li>
</ul>
</li>
<li>Local date/time filtering. Typically the current date and time is
not critical to the success of the test, so replacing the actual date and/or
time by a constant is common filtering operation. See for example DATE in
_filter() in
<a href="https://github.com/performancecopilot/pcp/blob/main/qa/866">qa/866</a>, or
TIMESTAMP in _filter__pmdumplog() in
<a href="https://github.com/performancecopilot/pcp/blob/main/qa/common.filter">qa/common.filter</a>
</li>
<li>Another approach to ensuring deterministic output for tests that
use PCP archives is to use the -z command line argument to most PCP commands
so that any date and time reporting uses the timezone of the archive
rather than the timezone of the machine where the test is being run.
See for example
<a href="https://github.com/performancecopilot/pcp/blob/main/qa/1072">qa/1072</a>.
</li>
<li>Remote host name, IP addresses, process ids (pids) and port numbers
are all examples of data that is likely to vary across the platforms
where the QA test scripts are run and so are candidates for filtering.
<a href="https://github.com/performancecopilot/pcp/blob/main/qa/100">qa/100</a>
includes several examples of this type of filtering.
</li>
<li>For some QA tests scripts the &quot;correct&quot; information
is confined to one or more fields or columns. If the other fields
and columns are not deterministic, then vertical filtering may be
appropriate.
For an example using cut see
<a href="https://github.com/performancecopilot/pcp/blob/main/qa/951">qa/951</a>
while
<a href="https://github.com/performancecopilot/pcp/blob/main/qa/957">qa/957</a>
shows how vertical filtering can be achieved using awk.
</li>
<li>Similarly, sometimes the information that is critical to
the outcome of a QA test may be in only some of the output lines
while the other lines are not constant across platforms.
Horizontal filtering may be pattern-based (e.g. using grep or sed)
or ordinal line number based (e.g. using sed or awk to remove or
keep lines N-M) or FSA-based (e.g. using awk to skip lines until
the first line of interest is found, then output the following lines
until the last line of interest is found).</li>
<li>Gratuitous white space differences for the same command across different
platforms is another common scenario.
See for example od output filtering in
<a href="https://github.com/performancecopilot/pcp/blob/main/qa/079">qa/079</a>,
or wc output filtering in
<a href="https://github.com/performancecopilot/pcp/blob/main/qa/1059">qa/1059</a>.
</li>
<li>For some tests, a value may be deemed &quot;correct&quot;
if it is in an expected range.
In these cases filtering is required to map the actual value onto
a constant literal value. For an example, see the _range() function in
<a href="https://github.com/performancecopilot/pcp/blob/main/qa/178">qa/178</a>.
</li>
</ul>
To the extent that filtering removes or rewrites what would otherwise
be the output of a QA test script, there maybe some difficulty in triaging
test failures if you have only the expected NNNN.out and the observed
NNNN.out.bad files.
Most QA test scripts will emit unfiltered output and diagnostics to aid
triage and
these are written to a NNNN.full file which is retained for inspection
after the test has been run.

<h3><a name="notrun"></a>Not run</h3>
If QA test script NNNN decides it cannot or should not be run on the
current platform, it should create a NNNN.notrun file.
Optionally NNNN.notrun contains the reason the test has not been run.

The convenience function _notrun() defined in 
<a href="https://github.com/performancecopilot/pcp/blob/main/qa/common.check">qa/common.check</a>
may be used to create the file, write the reason to the file and exit.

Some common reasons for a test to be &quot;not run&quot; are:
<ul>
<li>the test is platform-specific and makes no sense on other platforms</li>
<li>an optional PMDA that is required for the QA test script is not installed</li>
<li>a command or application or service that is required for the QA test
script is not installed or cannot be found</li>
<li>a Python or Perl module that is required for the QA test script is not installed</li>
<li>the QA test script requires a kernel, library, application or service
at some minimum release or version level</li>
</ul>

<h3>PMNS ordering</h3>
If a QA test script uses an application that traverses the PMNS
for a non-leaf metric name (usually from the command line or a configuration
file)
then depending on the PMDA(s) involved,
there is a chance that the names in the PMNS may be processed
in different orders on different platforms.
The simplest solution is to enumerate the PMNS nodes of interest
in the QA script using pminfo, sort the list and then have the
application being tested operate on the leaf nodes of the PMNS one
at a time.
<a href="https://github.com/performancecopilot/pcp/blob/main/qa/647">qa/647</a>
illustrates an example of this approach.

<h3>Instance ordering</h3>
Some PMDAs maintain their instance domains in a manner
that may present instances in non-deterministic order, so while
all instances may be present on all platforms, the sequence of the
instances within a instance domain or in a pmResult is not the same
everywhere.
The QA helper application $here/src/sortinst (source in
<a href="https://github.com/performancecopilot/pcp/blob/main/qa/src/sortinst.c">qa/src/sortinst.c</a>
may be use to re-order the reported instances so the sequence is
deterministic.
See
<a href="https://github.com/performancecopilot/pcp/blob/main/qa/1180">qa/1180</a>
for an example that uses $here/src/sortinst to re-order pminfo -f output.

<h3>Sort ordering</h3>
Unfortunately sort does not produce the same sorted
order on all platforms by default.
We need to take control of LC_COLLATE and the decision has been made
to standardize on POSIX as the collating sequence for PCP QA.
$LC_COLLATE is set and exported into the environment in
<a href="https://github.com/performancecopilot/pcp/blob/main/qa/common.rc">common.rc</a>,
but this is a relatively recent change so LC_COLLATE=POSIX is liberally
sprinkled throughout the QA test suite ahead of running sort.

<h3>Filesystem directory order</h3>
Some PMDAs use readdir() or similar routines to scan a directory
contents to find metrics and/or instances.
This practice is certain to expose platform differences as the order
of directory entries is unlikely to be deterministic.
Judicious use of sort will be required. See
<a href="https://github.com/performancecopilot/pcp/blob/main/qa/496">qa/496</a>
for a typical example of how this should be done.

<h3><a name="alternate.out"></a>Alternate .out files</h3>
For some PMDAs and/or some QA test scripts,
no amount of clever engineering will hide the fact
that a &quot;correct&quot; test execution on one platform may produce different
output to &quot;correct&quot; test execution on another platform.
In these cases the simplest choice is to not have a single NNNN.out
file, but rather have a set of them and the QA test script chooses the
most appropriate one and links this to NNNN.out when it starts.

Alternate NNNN.out files may be required in the following
types of cases:
<ul>
<li>PMNS differences (especially for the kernel PMDA), e.g
<a href="https://github.com/performancecopilot/pcp/blob/main/qa/012">qa/012</a></li>
<li>Base service differences, e.g. IPv6 may, or may not, be present for
<a href="https://github.com/performancecopilot/pcp/blob/main/qa/365">qa/365</a></li>
<li>Hashing differences</li>
<li>Different word size may mean different underlying metric data type
differences, e.g. in
<a href="https://github.com/performancecopilot/pcp/blob/main/qa/926">qa/926</a></li>
<li>Endianness differences make some dump outputs completely different,
e.g. in
<a href="https://github.com/performancecopilot/pcp/blob/main/qa/422">qa/422</a></li>
</ul>

</body>

${this.title}

Code Editor : portability-and-qa.html