[GUIDELINE] Testing Requirements & Protocols (Release Candidacy)

  • This forum entry will detail the necessary testing requirements and protocols to follow in order to qualify a release candidate for a full release.

    It is important to note that these guidelines MUST be followed to push any projects labeled as Release Candidate to RELEASE status. Failure to comply with these guidelines will result in either

    1. Project is rolled back, and access is restricted to publish any further releases.
    2. Temporary sanction from the development team (repeated offenses / major security flaw or issue)

    Testing Requirements & Protocols:

    In order for a release candidate to be considered for a release, the following criteria must be met:

    1. Release Candidate must be tested thoroughly with 24 - 36 hours of logged testing time. This is split into two sections: ACTIVE and IDLE. There must be at least 12 hours of active testing (in game tests), and 12 hours of idle testing (server running with no hindrance) minimum to satisfy this time requirement.
    2. If this candidate is marked as EXPEDITED (project patches a major exploit or other pressing issue), the candidate must be tested for a minimum of 12 hours, with a full report detailing the findings of said testing.
    3. If any severe issues, bugs, or exploits are discovered, any logged testing hours (and reports if eligible), must be concluded and subsequentially discarded
    4. Every time an issue is discovered, and the logged hours (and reports if eligible) are discarded, a new release candidate must be published with the appropriate fix and the testing procedures must be reconducted.
    5. If a release candidate otherwise fails to meet quality specifications, for reasons not otherwise specified, any logged testing hours (and reports if eligible) must be concluded and subsequentially discarded.
    6. When a release is published, the full testing log must be published with a detailed list of changes in a forum post, by the publishing author. If a report was also completed, the report must also be included. Both the tracked testing time and the reports if eligible must be included in PLAIN TEXT and not in the form of any media or attachment. These may be enclosed in spoilers, to reduce the visual length of the post.
    7. If a release is pushed to the live server and a severe issue, bug, or exploit is found during live time, the live server must be reverted to the previous release, the release with the discovered issue must be discarded, a new release candidate must be published with the appropriate fix, and the testing procedures must be reconducted.

    When logging testing time, the following format must be used:

    MM/DD/YYYY @ HH:MM UTC (START / END)

    The hours and minutes should be in 24 hour time.

    Here is an example of this format:

    Testing Time Log:

    ACTIVE:

    01/01/1961 @ 18:39 UTC (START)

    01/01/1961 @ 22:20 UTC (END)

    01/02/1961 @ 15:08 UTC (START)

    01/02/1961 @ 21:37 UTC (END)

    01/03/1961 @ 14:34 UTC (START)

    01/03/1961 @ 19:55 UTC (END)

    IDLE:

    01/01/1961 @ 18:39 UTC (START)

    01/02/1961 @ 22:20 UTC (END)

    Note: Logged testing hours that occur during another user's logged testing hours do contribute towards the overall hourly requirement, provided that all users currently testing are not testing repetitively.

    The goals with these guidelines are to ensure that not only are the releases we are pushing are validated as working, but evidence is provided, and transparency is shown when publishing the releases in the form of forum posts, to provide the community with a much clearer presentation of what is happening behind the scenes, alongside a sense of security knowing there is evidence proving that the projects have been verified as tested and working.

    Edited 7 times, last by Paldiu (September 14, 2022 at 4:18 PM).

  • Paldiu September 14, 2022 at 2:06 AM

    Changed the title of the thread from “[Guideline] Testing Requirements & Protocols (Release Candidacy)” to “[GUIDELINE] Testing Requirements & Protocols (Release Candidacy)”.