← Operations

Infrastructure Onboarding

Welcome to the team!!! Follow along for what to expect your first few days and weeks as part of our Digital Infrastructure team.

Week 1

Day 1

Ground Control helps with Macbook basics

The Ground Control team hosts a Monday morning call for new hires to go through and get the initial machine setup steps out of the way. They’ll familiarize you with where to look for corporate managed software and how to get the basics setup on your machine.

Meet the Team

Our teams will host a virtual lunch for introductions. An ongoing rotation of Pecha Kucha’s should be in the works, but this is a great chance to meet the rest of the folks on your team and hear a little bit about them.
noonish CST

Employment Documentation

The I-9 document should be completed during the first 3 days on the job. Depending on how much interaction is required by your manager we want to set aside some time to work through this.

Days 2-5

Corporate NEO

As of this writing the corporate teams host a new employee orientation for new hires that runs Tues - Thursday to give an overview of corporate systems, policies and processes.

Tools of the organization

We’ll work through the platforms available to all of the Banno/Digital groups before digging in later to the specific tools that each team uses. There’s some configuration that you’ll want to setup during this first week to make sure you’re getting the most out of the systems we all rely on.

Wiki

You’ll see quite a few references to docs.banno.com. This is our central point of organizational wide and quite a bit of our team based information. We’ll circle back around how to contribute and run this locally so you can contribute content as well. We’re midstream of moving our organization from a spoken word to a written word culture.

  • Site Reliability; incident processes
  • Departments; basic building blocks of Banno
  • Onboarding guides; basic guides to getting familiar with our stack.
  • Docs Day; quarterly initiative we use to keep pushing forward our culture of the written word.

G-Suite

Email filters

It’s going to be worth while to filter out a few of the email floods that we inflict upon ourselves. Google makes heavy use of labels and for this flow to work you’ll need two; one for “Approvals” and another for “Upgrades”. Create those labels first and then replace the place holder fields in the code block below and save into a new file with the .xml extension. Once saved you can navigate to your email settings > filter and import. This is really just a starting point and you can do much more interesting things in the future.

Be sure to replace the following placeholder fields before importing the filters.

  • $FIRST => Bullwinkle
  • $LAST => Moose
  • $EMAIL => bullwinkle.moose@bonno.com
  • $TAG => Approvals / Upgrades (two locations in the code block below)
<?xml version='1.0' encoding='UTF-8'?>
<feed xmlns='http://www.w3.org/2005/Atom' xmlns:apps='http://schemas.google.com/apps/2006'>
	<title>Mail Filters</title>
	<id>tag:mail.google.com,2008:filters:2881463829679875628,4976443312165359721,z0000001604346922301*2072399024897993517</id>
	<updated>2021-06-01T19:38:52Z</updated>
	<author>
		<name>$FIRST $LAST</name>
		<email>$EMAIL</email>
	</author>
	<entry>
		<category term='filter'></category>
		<title>Mail Filter</title>
		<id>tag:mail.google.com,2008:filter:4976443312165359721</id>
		<updated>2021-06-01T19:38:52Z</updated>
		<content></content>
		<apps:property name='to' value='approvers@banno.com OR accesschangeapprovers@banno.com'/>
		<apps:property name='label' value='$TAG'/>
		<apps:property name='shouldArchive' value='true'/>
		<apps:property name='shouldNeverSpam' value='true'/>
		<apps:property name='sizeOperator' value='s_sl'/>
		<apps:property name='sizeUnit' value='s_smb'/>
	</entry>
	<entry>
		<category term='filter'></category>
		<title>Mail Filter</title>
		<id>tag:mail.google.com,2008:filter:z0000001604346922301*2072399024897993517</id>
		<updated>2021-06-01T19:38:52Z</updated>
		<content></content>
		<apps:property name='subject' value='unattended-upgrades'/>
		<apps:property name='hasTheWord' value='list:(&lt;infrastructure.banno.com&gt;)'/>
		<apps:property name='label' value='$TAG'/>
		<apps:property name='shouldArchive' value='true'/>
		<apps:property name='sizeOperator' value='s_sl'/>
		<apps:property name='sizeUnit' value='s_smb'/>
	</entry>
   </feed>
Signature

We suggest a simple signature to be included on your emails. You’re welcome to be as concise or creative as you like.

$FIRST $LAST
$TITLE
$COMPANY (either Banno or Digital or Jack Henry)
Google drive

Download and setup Google Drive. The default path will be /Volumes/GoogleDrive/My Drive/ and Volumes/GoogleDrive/Shared Drives/. To make new machine setup easier we recommend setting up a few shortcuts(symlinks) to the .dotfiles that your shell uses to configure various services.

dotfiles

Note: This will overwrite any currently existing instance of the files being linked because of the use of the -f option. If you’ve already customized your “dotfiles”, such as your .zshrc, you will need to manually create the /Volumes/GoogleDrive/My\ Drive/dotfiles directory, and then copy your files into your Google Drive. After that, you should be okay to create the outlined below.

After installing Google Drive go to MacOS System Preferences > Security > Privacy to allow full disk permissions for terminal/iterm. Running the code block below from shell should get the basics setup.

  • touch creates an empty file
  • mkdir will create a new folder
  • ln -sf creates a symlink to the location. The symlinks should be in your ~/ home folder.
cd ~/
mkdir /Volumes/GoogleDrive/My\ Drive/dotfiles
touch /Volumes/GoogleDrive/My\ Drive/dotfiles/.digrc
ln -sf /Volumes/GoogleDrive/My\ Drive/dotfiles/.digrc .digrc
touch /Volumes/GoogleDrive/My\ Drive/dotfiles/.zshrc
ln -sf /Volumes/GoogleDrive/My\ Drive/dotfiles/.zshrc .zshrc
touch /Volumes/GoogleDrive/My\ Drive/dotfiles/.zsh_pass
ln -sf /Volumes/GoogleDrive/My\ Drive/dotfiles/.zsh_pass .zsh_pass
touch /Volumes/GoogleDrive/My\ Drive/dotfiles/.zsh_history
touch /Volumes/GoogleDrive/My\ Drive/dotfiles/.gitconfig
ln -sf /Volumes/GoogleDrive/My\ Drive/dotfiles/.gitconfig .gitconfig
touch /Volumes/GoogleDrive/My\ Drive/dotfiles/.gitignore_global
ln -sf /Volumes/GoogleDrive/My\ Drive/dotfiles/.gitignore_global .gitignore_global
mkdir /Volumes/GoogleDrive/My\ Drive/dotfiles/.kube
ln -sf /Volumes/GoogleDrive/My\ Drive/dotfiles/.kube .kube
mkdir /Volumes/GoogleDrive/My\ Drive/dotfiles/.govomi
ln -sf /Volumes/GoogleDrive/My\ Drive/dotfiles/.govomi .govomi
mkdir /Volumes/GoogleDrive/My\ Drive/dotfiles/.aws
ln -sf /Volumes/GoogleDrive/My\ Drive/dotfiles/.aws .aws
mkdir /Volumes/GoogleDrive/My\ Drive/dotfiles/.azure
ln -sf /Volumes/GoogleDrive/My\ Drive/dotfiles/.azure .azure
mkdir /Volumes/GoogleDrive/My\ Drive/dotfiles/.banno
ln -sf /Volumes/GoogleDrive/My\ Drive/dotfiles/.banno .banno
mkdir /Volumes/GoogleDrive/My\ Drive/dotfiles/.ssh
ln -sf /Volumes/GoogleDrive/My\ Drive/dotfiles/.ssh .ssh

From here you’ll want to add some basic configuration to each of these dotfiles.

.digrc

to filter out some of the noise when using dig from the shell

+noall +answer
.gitconfig

A VERY basic git config

[user]
	name = $FIRST $LAST
	email = $FIRST$LAST@users.noreply.github.com
[core]
	excludesfile = ~/.gitignore_global
[url "git@github.com:"]
	insteadOf = https://github.com/
.gitignore_global
# OS generated files #
######################
.DS_Store
.DS_Store?
._*
.Spotlight-V100
.Trashes
Thumbs.db
.zshrc

Your shell may behave strangely if google drive doesn’t start up correctly on boot. If you choose to use a local config file you’ll want to back up a scrubbed version of it somewhere in the cloud. Highly recommend something like ohmyzsh and iterm for your shell customization.

# Basics
# If storing history across nfs share(ie.google drive) a better method of handling file lock may be required
setopt HIST_FCNTL_LOCK
# stamp shown in the history command output.
# The optional three formats: "mm/dd/yyyy"|"dd.mm.yyyy"|"yyyy-mm-dd"
HIST_STAMPS="yyyy-mm-dd"
# Custom zsh history file path
HISTFILE="/Volumes/GoogleDrive/My Drive/dotfiles/.zsh_history"
# do not display a line previously found
setopt HIST_FIND_NO_DUPS
# do not write duplicate entries in the history file
# setopt HIST_SAVE_NO_DUPS
# remove extra blanks before recording history
setopt HIST_REDUCE_BLANKS
# share history between sessions
# setopt SHARE_HISTORY
# don't record entry that starts with space
setopt HIST_IGNORE_SPACE
## sourcing a password file. Helpful so that you can share your zsh config file without revealing passwords. 
source ~/.zsh_pass
# User Aliases
## quick directory navigation
alias envir='cd ~/code/src/github.com/Banno/environments'
alias wiki='cd ~/code/src/github.com/Banno/wiki'
alias github='cd ~/code/src/github.com/'
# random generator
# ex. rando 20
alias random_hash='head -n 4096 /dev/urandom | openssl sha224'
alias rando='openssl rand -base64 16 | colrm 17'

^ beyond that you’ll discover what you like and what is appropriate to the task. You can rabbit trail on this all day online or ask some team mates what they’ve found most helpful to add to their shell config file.

Calendars and Groups

Ask a team member to help setup access to the following.

  • PTO calendar
  • Google Groups - we use these as basic email distribution lists and for granting permissions. The membership should be set up in a nested manner so you’ll inherit overarching access granted by larger lists. ie. Staff > Engineering > Infra Team.

Office 365

Slack

Slack can be the wild west with the sheer number and names of channels. We find this Communication Guide to be of great help understanding and mapping everything out. Broader channels worth mentioning: org-faq, org-new-hires, office-remotes Domain specific channels:org-deployments, org-reliability-ff Team Specific: org-infrastructure, org-pulsar, org-engineering, alerts-infrastructure-production,alerts-infrastructure-lower-envs

Microsoft Teams

Microsoft Teams is really a bit of a footnote in Banno, but Jack Henry and the Digital team use it across all other divisions so it is expected we’ll operate in both worlds as our duties require.

  • Cross Org Collaboration (search docs.banno.com for instructions) run through of basic Digital channels

GitHub

All of our code is in Github and a good bit of our access is assigned by group membership in GH. We’ll want to make sure you’re basic git config is setup and that you have the appropriate team membership.

Jira

The go to place for picking up tasks and seeing what is in flight. Here is a link to our roadmap

  • Verify infra team membership and access.
  • point to org-jira and the pinned jira guides. Specifically how GH/Jira require specific syntax to pull details across from one to the other.

PagerDuty

Set a calendar invite for 45 days in to introduce into PagerDuty rotation. Verify account has been setup and show notification waterfall. university.pagerduty.com for 100 level fundamental courses.

Security

  • personal password mgmt local: keepassx online: lastpass/onepass additional layers: yubikey 2fa - The banno app uses authy,
  • org-security for larger security concerns or risks.

Week 2-3

Days 6-21

Access

  • double check that they are included in the Infra Google Group
  • double check that they are included in the Slack @infra mention
  • add to Infra team on Jira
API Access
  • Azure - double check production Azure access by looking up AD group membership
  • AWS - double check account has been created
  • Vsphere access - make sure they have rights and can run govc
  • Artifactory - make sure they have access

Tools of the team

Homebrew

You will need Homebrew installed to grab a common base of packages that all infrastructure team members use.

Dotfiles repository

We have a repository called dotfiles which we use to install a base set of tools on all of our Macs using a custom Homebrew tap/cask. You should clone this repository down and run make inside of it.

Note: the above make command will fail if the HOMEBREW_BANNO_ARTIFACTORY_TOKEN environment variable, as defined in the “artifactory” step, isn’t available in your shell session. It can also fail if you’ve already installed packages through another non-homebrew method, such as using a .dmg installer directly from a website. You can find your token by clicking on the top right of your Artifactory profile and clicking “Edit Profile”.

Tool configuration

The following environment variables will need to be set in your environment. If you followed one of the previous sections and used the ~/.zshrc template, your shell will automatically try to locate the ~/.zsh_pass file. This file is where you can store passwords that will be used often for API access, as it’s not symlink’d to your Google Drive. Note that this method is saving plaintext passwords in unencrypted storage. A password manager is still advised to store passwords, but integrating a password manager to work with environment variables is out of the scope of this onboarding documentation.

Datadog

Artifactory (Attempt to login once using the artifactory link. After that, post in #org-ground-control to request to be added to the Infrastructure group in Artifactory, unless someone from our team is already helping you through this step.)
  • Navigate to our Artifactory instance
  • Log in (either a Google login will appear, or you may see a “SAML SSO” button on a login form)
  • Click your email at the top right of the page > “edit profile”, or click this link
  • Click the cog icon to generate a new API key, copy it
  • Replace $token in the below snippet to be added to your ~/.zsh_pass file
export HOMEBREW_BANNO_ARTIFACTORY_TOKEN="$token"
Azure

Set your default subscription to be Stg_Digital_Banno

az account set -s Stg_Digital_Banno
Banno Credentials
export BANNO_LDAP_USER="<your LDAP username>" # firstnameinitiallastname@jackhenry.com
export BANNO_SSH_USER="<your SSH username>"
export BANNO_SSH_KEY="${HOME}/.ssh/id_rsa"
AWS (skippable if you don’t have credentials yet)
# AWS Credentials
export AWS_ACCOUNT_ID="423417858957"
export AWS_ACCESS_KEY_ID="<your key>"
export AWS_SECRET_ACCESS_KEY="<your secret key>"
export AWS_DEFAULT_REGION="us-east-1"
vSphere Credentials (skippable if you don’t have credentials yet)
export VSPHERE_SERVER="10.203.131.24"
export VSPHERE_USER="${BANNO_LDAP_USER}@jkhy.com"
export VSPHERE_PASSWORD="<your vSphere password>"
Terraform Credentials

This step just aliases credentials already set elsewhere to variables terraform can understand. If you skipped previous steps, you may have trouble running terraform {init|plan|apply}s.

export TF_VAR_vsphere_user=$VSPHERE_USER
export TF_VAR_vsphere_password=$VSPHERE_PASSWORD
export TF_VAR_vsphere_server=$VSPHERE_SERVER
export TF_VAR_ssh_user=$BANNO_SSH_USER
export TF_VAR_private_key=${HOME}/.ssh/id_rsa
export TF_VAR_public_key=${HOME}/.ssh/id_rsa.pub
export TF_VAR_aws_access_key_id=$AWS_ACCESS_KEY_ID
export TF_VAR_aws_secret_access_key=$AWS_SECRET_ACCESS_KEY

Kube Config

Copy the kubeconfig file here to your ~/.kube/config

Do you have a preferred code editor?

  • VIM
  • emacs
  • Visual Studio Code

Blackbox

Follow the instructions on how to set up Blackbox here.

Env-setup

env-setup - how to standup a local environment for testing. Work with a team mate to run through our environment setup tools.

Obsidian

Download the Obsidian client, and clone down our Infrastructure knowledge repository, which holds all our infrastructure-related documentation

Sign your commits with GPG

git config --global commit.gpgsign true

SSH Key

SSH Config File Structure and Patterns

Follow the steps here to create a SSH config file if it does not yet exist in your computer. After you’ve added your ssh keys to the banno-user recipe, it’ll allow you to ssh into our remote servers.

  AddKeysToAgent yes
  UseKeychain yes
  IdentityFile ~/.ssh/id_rsa
  User firstinitiallastname
  StrictHostKeyChecking no
  ForwardAgent yes

You should now be able to ssh into our servers. Try running ssh mesos-agent0-lks.uat-2.banno-internal.com in your terminal.

Retrospectives

Invite to and verify Retrium access for the team’s retrospective, which happens every other week.

Org wide expectations and operation

Values and Principles
Career Ladder
The Monthly Customer Meetup is a great chance to see what Banno is working on and hear back from our customers. Facilitated by department directors.
Digital All Hands; tend to shift around and be every other month. Facilitated by Ben Metz
Engineering All Hands; tend to shift around and be every other month. Facilitated by Chad Killingsworth
Services Engineering All Hands; ~every other month. These oscillate between svcs eng and infra or svcs eng and mobile eng. Facilitated by service eng team leadership.
Conferences! are one area of learning that we want to make sure as many people can leverage as possible. A culture of continuous learning is key to the life of our organization. We can walk through the basic mechanics and overall budgeting process for this to give you a better idea of what to keep in mind when picking which conference you want to go to.

Energy level discussion by Ben - https://drive.google.com/drive/u/0/folders/0B28feglletdRZVBVUk9nQmlEWTQ
Organizational change overview - https://drive.google.com/drive/u/0/folders/0B492P2w7cQLtTzFVZVlWNXpWR1k

Familiarizing with workflow

Roadmap Overview
Let’s walk through a basic flow of how to manage Jira details to make sure our work is visible to the rest of the organization and be pulled into the overall Banno Roadmap. Jira Board flow
Domain Ownership is a map to which technical domains are owned buy which teams.

Team expectations and operation

We’ve got a number of processes and expectation guides in varying states of development. They make up the basic building blocks of how we aspire to function as as a group. We can discuss a few and you can begin slowly reading through these guides.

Basic Meetings

Infra Sweeps (weekly) Retrospectives (every other week) Pairing (ad-hoc) Mobbing (some planned some ad-hoc) Tech Talks (a few a month) 1-1s (monthly)

Aspirational

As these meetings have come into being we typically have run them every other week for 3-4 iterations and then move them to quarterly revisit of the topic. Infra Manifesto|How we aspire to work
Agile - to make sure we’re operating under similar sets of assumptions and principles for software development Team Boundaries and Bonding - as our infrastructure teams have split up responsibilities we still want to keep a close connection. We’ve done virtual tea tastings, anime
Buddy System - a system that we’ve setup to provide some guardrails for one another.

Week 4

Now that you’re a few weeks into the belly of the beast it might be time to review the sorts of tools that are helpful to take it all in.

Productivity tools

Week 5

Learning

Podcasts

  • GoTime (golang) - deep backlog
  • Exponent (about tech and society) - large backlog/library of ~180 shows. Best enjoyed one or two a week

Internal

  • #sig-mudcow - general discussion around showing off something you’re working on / have worked on
  • #sig-language channels:
  • team exercises; we’ve used Exercism in the past to help bring our teams together and build them up around particular technologies. Let’s see what we have in motion currently then see what we could get into the works.

Week 6

Errata

Reading

Eventually

  • Discuss being added to the on-call Pagerduty schedule