Software Development Life Cycle (SDLC) Checklist

March 2023

Project Initiation and Business Planning

Request reasons for change or feature

1. Regulatory requirement
2. Feature request to meet contract obligations
3. Software bug
4. Software refactor to improve readability and updatability of code
5. Technology changes
6. Security updates
7. Deployment or process improvement
8. Feature addition

Document change request

Changes are communicated and documented in a shared document and project management software. The change request is entered into project management software which can link to scope, technical, design and functional docs.

Standard project documents

Most projects should complete the following set of documentation

1. Product Brief - a description of the problem
2. Project Analysis - a summary of the project analysis performed. This will live in it’s own folder within the project folder. Varies by project and analyst.
3. Product Requirements - Requirements, scope and a definition of project success
4. Design Doc - design requirements
5. Implementation/Notes - documentation of decisions and truths that describe how the requirements were implemented
6. QA - a confirmation that requirements have been met. Various methods may be used to demonstrate the successful implementation of the change request/feature. E.x. test cases, video, live demo, database query, conference call
7. Operations Doc - details onboarding requirements for Implementation team
8. Deployment Plan - Optional. Organizational list of non-standard deployment requirements. When not provided, a standard deployment scenario, documented below, is expected.

Approve change request

Feature scope and functionality are approved by management. Once approved, the project is placed in a backlog or prioritized in the current roadmap.

The role of the Project Manager

The Project Manager’s role is to verify the project maintains good adherence to the process defined here. This role will track and monitor project risks, open questions, security assessments, communication channels, dependencies, timelines and overall project progress. The Project Manager performs many tasks outside of the standard project process such as financial forecasts and cost tracking, capitalized work and its reporting, validating development processes against Corporate and Federal standards, and helping to maintain an acceptable project plan that meets business priorities and timelines.

Project Design and Discovery

Project exploratory meetings

The initial discussion around the project. Includes a small group of leads to identify the problem, validate scope, dependencies and technical feasibility.

As of April 2021, the exploration meeting for each project has been replaced with a recurring meeting that will discuss all upcoming projects that require an exploration period. The development manager is responsible for the scheduling and agenda of the exploration meetings.

Project discovery meetings

Recurring discovery meetings are scheduled by the Engineering or Project Manager and include the product owner, product designer and selected engineers - a cross-functional group of stakeholders. A product brief, project scope and design are completed and agreed upon. Project technical feasibility is completed and API changes are discussed and documented. Any integrations are tested and validated. Technology risks are identified and discussed. Security concerns are identified and solutions are approved by the security team. A high level estimate is agreed upon and communicated. The roadmap is updated to reflect any changes in project deliverables or timelines. A chat channel is created for the project and the stakeholders are invited. The channel’s topic will contain a link to the Product’s project folder and a description of the reason for the channel.

Project approval

Once scope and design are completed, project approval is given from the Software Engineering Director and the Technical Product Manager. The project is then added to the roadmap.

A project kickoff meeting is performed. This kickoff includes stakeholders, engineers, designers, the product owner, project manager, development manager, infrastructure and support. We will read through the entire Project brief, scope, requirements and design to bring all parties to the same level of understanding and enablement with a goal to understand what do we need to do to enable a user to meet the project goals.

Project backlog creation

From the scope and design, engineering construct a project backlog of Issues or a Project in the project management software that implement the design specifications and requirements. Multiple backlogs can be generated based on the number of development teams involved.

Project Planning

Planning Release and resource planning is scheduled on a reoccurring basis and includes the Software Engineering Director, the Engineering Manager, the Technical Product Manager, product owners and team leads. Prioritized projects that are ready for development (brief and scope completed) and prioritized are assigned engineering resources. Probable release dates are calculated and the road map is updated.

Change management

All project changes are approved and documented in discovery meetings, chat or email communications.

Project Development

Engineering development

All engineering teams are self organized and responsible for task backlog, intra-team assignment, schedule and feature development.

A common arrangement for a cross-functional project team is to first define and code the project API requirements which involves service teams and an analyst. Once complete, the project designer and client teams are enabled to integrate with the new API contract.

Project meetings

All running projects meet in sweep and standing meetings. Sweeps include the engineers, product owner, designers, documentarians and development manager. Sweeps are held to build consensus, realign priorities, remove blockers, track progress.

Standing meetings are an extension of the discovery meetings held during project definition. These meetings are cross-functional and exist to continue define the solution and react to changing requirements.

Coding and unit testing

All code is to be merged in through a pull request in source control. Code must be adherent to proper style and syntax. The code should be properly tested and verified to ensure addresses the product change or feature is implemented correctly.

Code review and merge

Note: this section only covers the repositories for library and service source code. Examples include banno-propagation (library) and banno-conversations (service). Deployment repositories are covered below in the Deployment And Release section.

• All pull requests require approval from at least 1 other human before being merged into the main/release branch.
• The approval must be completed by a different person than the pull request author. “Author” refers to the individual who opened the pull request.
• Repository owners are allowed to require pull request approvals from more than one person.
• Merging a pull request is considered implicit approval.
  • For example, if Employee A or an automated process opens a Pull Request then Employee 2 merges it the code change is considered approved by Employee 2.
• Auto-merging of PRs is NOT allowed. A human must approve every PR. Bot-created PRs are allowed (for instance, to bump dependency versions) but they are subject to the same review processes as if a human opened the PR.
• In an urgent situation (hotfix / incident) a PR can be merged without approval. But the individual who merged it must request and receive retroactive approval as soon as reasonably possible following the guidelines listed above.
  • Github’s UI does not show the “approval” button post-merge, so retroactive approval is granted by adding a comment to the PR. Various terms are used to indicate approval such as “LGTM”, “approved”, “✅”, etc.

Prepare for deployment

All merged code will be deployed to an intermediary device or location to be additionally QA and beta tested before being released to Production.

Retrospective

A meeting to allow all parties who contributed to the project a forum to provide feedback to better improve how projects are executed.

Quality Assurance

QA definitions

QA actions are defined and enforced by the Engineering Manager, as well as the design, analyst and product teams to verify all project requirements are met.

Integration

All integrations are unit, UI, manual and/or beta tested to ensure proper coding.

Regression

All new code requires proper and correct regression testing to ensure functionality and features are maintained.

Design acceptance

Product designers will validate all new code that will affect how users view and use the product.

Security

During discovery or design of feature development, a decision should have been made on whether a Security Assessment is required. The security team is actively engaged with all security decisions made by technologists. The security team will decide whether to engage the JHA Software Security Group for a Security Assessment engagement.

If the security team was engaged on the project, a member of the team must validate any security solutions identified during discovery before deployment.

Performance

Proper performance tests will be identified and performed by product, service and infrastructure teams. Basic levels of performance will be met.

Deployment and Release

Services: Updating test environments

Once a feature or bug fix is merged, tests are run. If successful, the new code is automatically pushed to the lowest environment. Services can choose to automatically deploy to the next test environment either automatically or manually on a service-by-service basis.

Deployment approval for Banno

To release a Banno service or infrastructure change to production there are multiple pipelines to track and deploy changes.

Code review and merge rules apply to all the deployment pipelines described below. Some pipelines impose additional requirements.

Jenkins pipelines (deprecated)

Review/approval addenda:

Commits are monitored from the environments Git repository, and a change log is generated. This change log contains the title, author and URLs of Pull Requests and commits included in the release. Release notes are generated from the change log and communicated to approvers@banno.com via email. To be able to approve changes sent to approvers@banno.com, a user must be in the deployapprovers@banno.com group. Membership to the deployapprovers@banno.com group is updated as necessary to reflect staff changes. Removals and additions are communicated to the approvers@banno.com list for approval from someone the deployapprovers@banno.com. An incident requiring a hotfix (or similar time-sensitive deployment) could bypass the approval and ask for it retroactively.

Terraform Pipeline (Github Actions)

A pull request created in the banno-infra terraform repository autogenerates a “change request” in ServiceNow for production.

Argo Pipeline (kubernetes)

To release a service to production via Argo, changes are made through the banno-apps Git repository. There are two primary ways these changes are made:

1. Automated Pull Requests

   An automated pipeline creates pull requests to update the version of a deployed service. This pipeline is started in the repositories for each service in response to a code change for the service. These pull requests include a changelog in the description that outlines the changes being deployed. A “change request” in ServiceNow is also created upon pull request creation for production.

2. Manual Pull Requests

   Employees may also manually open pull requests for operational changes. These manual PRs typically do not include a service changelog. A change request in ServiceNow is also created upon pull request creation for production. Examples:

   • rolling back to a previous version
   • updating environment variables
   • adjusting CPU/memory resources

Review/approval addenda:

• There is a special github group DeployApprovers. Members of this group are considered codeowners of all services in banno-apps and can therefore grant approval to any service.
  • This is typically required only in urgent situations when the regular team or eligible codeowner is unavailable to provide approval.
• For Production deployments the approver must NOT be listed on the service’s change log.

Deployment approval for Digital Core

To release to production, all pull requests to:

• the k8s-apps repository after the point when this pull request has been merged will require an approval from the @JHADigitalCore/manager-approvals group.
• the environments repository, after this pull request has been merged will require approvals from the @JHADigitalCore/manager-approvals group.

The @JHADigitalCore/manager-approvals is maintained by those with “Maintainer” permissions on the member list, or those which have “Owner” permissions on the JHADigitalCore GitHub organization. The .policy.yml file, located at the root of the repository, designates only files located at a specific directory path will require an approver by one member of the @JHADigitalCore/manager-approvals group. An incident requiring a hotfix could bypass the approval and ask for it retroactively through other means besides a pull request review.

Production Configuration Changes

Configuration changes to production systems can only be applied by members of the infrastructure team. These changes could be made in response to changing traffic patterns, system resource usage or service scale for an incident or in the due course of daily operations. In the case of an incident a change could be made either to the Marathon or Kubernetes control plane directly or for daily operations within the environments repository to be later applied by automation. If a change is made directly on the control plane then the environments repository will be updated to avoid configuration drift.

These changes could include, but are not limited to the following

• Resources (Example: CPU, Memory, Disk, Replicas, etc.)
• Environment variables (Example: USE_HTTP: true -> USE_HTTP: false)
• Image version number (Example: v0.0.1 -> v0.0.2)
• Configuring kernel parameters (Example: sysctl values)

Databases

For database changes, both configuration and permission changes are deployed the same way other production configuration changes are made. Only applied by members of the infrastructure team and through the same approval process.

Secrets for Digital Core

For secrets stored in HashiCorp Vault, development teams have write-only access to write secrets within their respective paths. This reduces the number of eyes in the secret’s chain of custody. Vault uses versioned secrets to allow a history of changes to be persisted.

Security Hardening

Kubernetes: Rely on official images more so than unofficial If not using an official image, Banno will evaluate the source Dockerfile of the image prior to allowing its use. Banno will run a bare bones image for the current task at hand, augmenting only as necessary with CA certificates and additional components. Images are downloaded and re-tagged then republished to internal artifactory storage so the image stays unmodifiable. A reboot daemon will perform safe automatic node reboots when indicated by the package management of the underlying OS. This daemon is used to facilitate safe reboots across the cluster and thus bring more OS security to the Kubernetes cluster.

VMs: Banno utilizes an industry standard declarative Infrastructure as Code tool to create virtual machines. After the infrastructure has been scaffolded, Banno will utilize a configuration management tool to install and configure a standard base of software. On top of the standard installation, the machine is augmented with only the required software to perform its task.

In this standard base of software, Banno will configure unattended upgrades and post provisioning process setup the SSH daemon configuration; with a list containing public keys of users in the Infrastructure group.

Setup Logging configuration

Add Antivirus and its accompanying configuration. The scan runs on a schedule and notifies the infrastructure team of any findings or errors. Disable the root account.

An administrator’s SSH key pair is generated locally and then the public SSH keys are committed to Git, private keys are stored locally on encrypted storage. The public keys managed in Git are applied to all systems to control administrative access via SSH.

Encrypted transit is required for administrative access to manage systems. Each user is assigned a security policy that determines access. Two factor authentication is required in order to connect.

Patching and upgrades are performed automatically and unattended by scheduled cron jobs. A list of applications is maintained in order to be released in a scheduled fashion. eg. PostgreSQL. Any applications that require a node reboot notifies the infrastructure team of the update’s availability and require a scheduled release process to ensure stability.

Perform deployment

The deploy pipeline tools deploy to production after approvals are acquired. The Pipeline and Core Cloud Infrastructure teams are able to deploy manually if there is an issue with the automated pipeline.


If a rollback is required, pull requests are created, reviewed and merged to the environment directory to decrement version numbers of the deployables. The standard or hotfix path is then followed for production deployment.

Operations approval

Operations is notified by the development manager of project completion. Operations will review the Operations doc which was generated during the development phase. The Operations doc is approved by the Operations Manager who is confirming that doc contains the needed information to configure and support the new feature.

A feature project will not be considered “Done” until Operations confirms they can support and implement it.

Mobile release

A release candidate is tagged in source control and the deployable binaries are generated for each client and uploaded to artifact storage or directly to the app store. Release notes that describe what is included in this release are generated and sent to the approvers group. The approval group must give one approval before the Mobile Build and Release team upload and install the new apps to the app stores. Once the apps pass review from the app store, they are released. Internal tools are used to help automate this deployment process.

New mobile versions, once available in the app store, can be installed on user’s devices. If the release is designated a critical update¹, a minimum version is set for the release and users are required to upgrade before using the app.

Customer Beta Testing

Enabling beta testing

All product features are enabled/disabled from a configuration database. Specific financial institutions and users can be granted newly released features for beta testing before enabling for all users at a financial institution.

Footnotes

Critical Update

(1) A critical update is defined as a widely released fix for a specific problem that addresses a critical bug or exploit/security vulnerability. The product owner, development lead and engineering director will determine if the next release is a critical update. If the release is considered critical then the minimum app version will match the current release version which will force the user to upgrade to the latest version.