Project Initiation and Business Planning
Request reasons for change or feature
- Regulatory requirement
- Feature request to meet contract obligations
- Software bug
- Software refactor to improve readability and updatability of code
- Technology changes
- Security updates
- Deployment or process improvement
- Feature addition
Document change request
Changes are communicated and documented in project management software. The change request is entered into project management software (Gitlab) which can link to scope, technical, design and functional docs.
Standard project documents
Most projects should complete the following set of documentation
- Product Brief or Scope - a description of the problem available in project management software
- Project Analysis - completed by Product Owner and not formally documented in a specific format
- Product Requirements - available in project management software
- Design Doc - design requirements - nothing formal today
- Implementation/Notes - documentation of decisions and truths that describe how the requirements were implemented (found in Gitlab)
- QA - no formal documents - Slack messaging covers communications - a confirmation that requirements have been met. Various methods may be used to demonstrate the successful implementation of the change request/feature. E.x. test cases, video, live demo, database query, conference call
- Operations Doc - minor release notes are generated. Monthly meeting to review release support.
Validate and research change request
The Product Owner and Engineering Manager validate change requests. Discussions with the customers further define feature scope, risk, interest and usability. The Support team also gives priority to identified customer issues that require a project to solve.
Approve change request
The Product Owner and Engineering Manager agree on project priority and approve change requests.
Project Design and Discovery
Project exploratory meetings
Daily standing team meetings will introduce change requests decided by Ben and Jordan.
Project discovery meetings
Performed in standing meetings including ad hoc breakout meetings.
Project approval
The Product Owner and Engineering Manager approves the project schedule
Project backlog creation
From the scope and design, engineering construct a project backlog of Issues or a Project in the project management software that implement the design specifications and requirements.
Project Planning
Planning Release and resource planning includes The Product Owner and Engineering Manager. Prioritized projects that are ready for development and prioritized are assigned engineering resources. Probable release dates are calculated and the road map is updated.
Change management
All project changes are approved and documented in standing meetings, chat or email communications.
Project Development
Engineering development
All engineering teams are self organized and responsible for task backlog, intra-team assignment, schedule and feature development.
Project meetings
Scheduled as needed by the Engineering Manager
Coding and unit testing
All code is to be merged in through a pull request in source control. Code must be adherent to proper style and syntax. The code should be properly tested and verified to ensure addresses the product change or feature is implemented correctly.
Code review and merge
All pull requests require code review from two employees, the Engineering Manager or Engineering Team Lead before being merged into master. The review must be completed by a different person than the pull request author. Hot fixes to alreasdy created releases still require 2 approvals and are only merged by Engineering Manager and Engineering Team Lead.
Engineering teams have implemented an automated task to automatically identify and update open source libraries that are out of date. This process does not bypass engineering approvals on the pull requests it generates. Test suites in lower environments are run to validate library version.
Prepare for deployment
All merged code will be deployed to an intermediary device or location.
Retrospective
Open Anywhere has an informal environment to allow all parties who contributed to the project a forum to provide feedback to better improve how projects are executed.
Quality Assurance
QA definitions
QA actions are defined and enforced by the Engineering Manager and the Product Owner. QA team verify all project requirements are met.
Integration
All integrations are unit, UI, manual and/or beta tested to ensure proper coding.
Regression
All new code requires proper and correct regression testing to ensure functionality and features are maintained.
Design acceptance
The Product Owner, Engineering Manager, and QA will validate all new code that will affect how users view and use the product.
Security
Engineering manager takes the security question to security team at Banno
Performance
Proper performance tests will be identified and performed by development teams. Basic levels of performance will be met.
Deployment and Release
Services: Updating test environments
UX and service CRs are packaged together, built manually, bundled and deployed. All current development will be pushed into the next minor version. Packaging and versioning are defined in a cron job that works with GitLab. Engineering Manager and Engineering Team Lead can approve and merge PRs added to released versions.
Minor release notes are built from a change log. QA does not sign off on releases but reviewed published versions.
The deployment pipeline requires unit tests, db integration tests (Flyway), static analysis (Spotbug), code quality scanning, dependency scanner to complete and pass before a minor version can be considered validated for deployment.
Once a feature or bug fix is merged, environmental tests are run. If successful, the new code is automatically pushed to a staging environment. A task to promote code to UAT can be executed. The binary versions on staging will be deployed to UAT as well as database updates. For services, an extra suite of core tests are run after a UAT deployment. This helps to determine whether a production build is possible.
Deployment approval
Engineering Manager and Engineering Team Lead can approve and merge PRs added to a released version.
Production Configuration Changes
Configuration changes to production systems can only be applied by members of the infrastructure team. These changes could be made in response to changing traffic patterns, system resource usage or service scale for an incident or in the due course of daily operations. In the case of an incident a change could be made either to the Marathon or Kubernetes control plane directly or for daily operations within the environments repository to be later applied by automation. If a change is made directly on the control plane then the environments repository will be updated to avoid configuration drift.
These changes could include, but are not limited to the following
- Resources (Example: CPU, Memory, Disk, Replicas, etc.)
- Environment variables (Example:
USE_HTTP: true->USE_HTTP: false) - Image version number (Example:
v0.0.1->v0.0.2) - Configuring kernel parameters (Example: sysctl values)
Databases
For database changes, both configuration and permission changes are deployed the same way other production configuration changes are made. Only applied by members of the infrastructure team and through the same approval process.
Security Hardening
Kubernetes: Rely on official images more so than unofficial If not using an official image, Banno infrastructure team will evaluate the source Dockerfile of the image prior to allowing its use. Banno will run a bare bones image for the current task at hand, augmenting only as necessary with CA certificates and additional components. Images are downloaded and re-tagged then republished to internal artifactory storage so the image stays unmodifiable. A reboot daemon will perform safe automatic node reboots when indicated by the package management of the underlying OS. This daemon is used to facilitate safe reboots across the cluster and thus bring more OS security to the Kubernetes cluster.
VMs: Banno utilizes an industry standard declarative Infrastructure as Code tool to create virtual machines. After the infrastructure has been scaffolded, Banno will utilize a configuration management tool to install and configure a standard base of software. On top of the standard installation, the machine is augmented with only the required software to perform its task.
In this standard base of software, Banno will configure unattended upgrades and post provisioning process setup the SSH daemon configuration; with a list containing public keys of users in the Infrastructure group.
Setup Logging configuration
Add Antivirus and its accompanying configuration. The scan runs on a schedule and notifies the infrastructure team of any findings or errors. Disable the root account.
An administrator’s SSH key pair is generated locally and then the public SSH keys are committed to Git, private keys are stored locally on encrypted storage. The public keys managed in Git are applied to all systems to control administrative access via SSH.
Encrypted transit is required for administrative access to manage systems. Each user is assigned a security policy that determines access. Two factor authentication is required in order to connect.
Patching and upgrades are performed automatically and unattended by scheduled cron jobs. A list of applications is maintained in order to be released in a scheduled fashion. eg. PostgreSQL. Any applications that require a node reboot notifies the infrastructure team of the update’s availability and require a scheduled release process to ensure stability.
Perform deployment
The Ground Control, Launch Control and Banno Infrastructure team members are enabled to deploy to production from their computers.
If a rollback is required, pull requests are created, reviewed and merged to the environment directory to decrement version numbers of the deployables. The standard or hotfix path is then followed for production deployment.
Operations approval
Operations is notified by the Engineering Manager of project completion. Operations will review the Operations doc which was generated during the development phase. The Operations doc is approved by the Operations Manager who is confirming that doc contains the needed information to configure and support the new feature.
A feature project will not be considered “Done” until Operations confirms they can support and implement it.
Customer Beta Testing
Enabling beta testing
All product features are enabled/disabled from a configuration database. Specific financial institutions and users can be granted newly released features for beta testing before enabling for all users at a financial institution.
Footnotes
Critical Update
(1) A critical update is defined as a widely released fix for a specific problem that addresses a critical bug or exploit/security vulnerability. The product owner, development lead and engineering director will determine if the next release is a critical update. If the release is considered critical then the minimum app version will match the current release version which will force the user to upgrade to the latest version.