Interviewing in IT – Finding Solid Candidates

One challenge within all IT organizations is finding and interviewing candidates to ensure that those hired into an organization will not only bring necessary skills, but also allow the organization to grow and evolve. Interviewing methods vary from company to company based on a variety of factors including culture, past experiences and human resource department influences. I hope to explore some factors I believe lead to successfully interviewing of candidates.

In my mind, there are four primary purposes of interviews:

1. Determine if the candidate has the proper technical skills to be successful. This includes a combination of technical knowledge and past experiences.
2. Determine if the candidates' personality is compatible with both existing team members as well as the company culture.
3. Determine if the candidate has the necessary willingness to learn new skills and learn from their team to evolve and grow in the position.
4. (Sometimes) Determine if the candidate has specific domain knowledge or industry connections that can only be obtained by hiring from outside the organization.

Here are the key items I keep in mind when interviewing, as well as encourage others to think about when interviewing candidates for my team and other teams I work with.

How Long?
Don't focus on how long the individual will be at the company; assigning a time frame will defer focus from the more critical aspects of the candidates' place within the organization. Focus instead on what the individuals career goals are. Can those be met at this company? What types of technology do they want to work on and learn? If these are technologies your company uses and will be using, the chances of a long term candidate increase greatly.

Technology is not the only factor, focus on other aspects of the individuals career goals. Are they expecting promotions into management or technical lead positions? These types of advancement opportunities will determine how long the candidate will be at the company.

2-way Street
I have been to many interviews that the individual conducting the interview forgot that interviews are 2-way streets. While it is important for a company to determine if an individual will perform well within the company structure and culture, it is equally important that the individual get a realistic feel for the company and see the company in a good light. You don't want to risk loosing a strong candidate because the focus was too much on asking the individual questions, and not enough on allowing them to see the possibilities your company has to offer.

Brain teasers are fine, but.....
Many organizations today use brain-teaser type questions to understand how a candidate solves problems and their methods for thinking and responding to stressful situations. These tests have a valid place within interviews, but need to be used properly to ensure you are getting valid results from the tests. It is important to only ask brain teasers that the individual will have the domain knowledge to approach answering. I have been in many Network Architect interviews and asked questions about algorithms around managing large datasets. This was not a good use of time for me or the interviewer. It is important to ensure that even if the candidate can not answer the brain-teasers, they have enough base knowledge to share their through process for solving the puzzle.

Personality Tests
Personality Tests tend to fade in and out of popularity as an interviewing tool. Many companies argue that they provide a glimpse into the candidates tendencies and habits and can allow the human resources department to determine their compatibility with existing personality types at the company. The problem is that many of these tests are quite easy to “study” for and this can heavily skew the results. My feeling is that taking the candidate to lunch or coffee with a small group of team members will show much more then a standard test ever will. The key with an interview is to make sure both parties are comfortable enough that they communicate as they would in any standard work situation.

By conducting the interview in places other then your standard conference room, you have a better chance of understanding what the individuals work habits are and how they converse with others. It is easy to watch how a candidate talks with folks at the office; it is an entirely different view to see how they interact with the wait staff at a restaurant and a very useful data point on their personality.

Work Location
“Remote” working is all the rage these days, especially in IT. You know my thoughts on it from previous postings, I think that it has it's place, but not all positions can function effectively while being isolated at home. Deciding if a position is going to be “remote” should be done before interviewing potential candidates, this decision should be based on the type of work this position will be doing, as well as that of the team and how well that work can be done in a distributed fashion.

Not only does work location include being “remote” or not, it also includes possible relocation of the candidate. It is important to set expectations up front on the companies' policies for relocation. If there is no relocation budget and the candidate is three-states away, it is probably not worth pursuing.

Finally, it is important to understand an individuals' constraints around relocation, both time frames and potential locations. If a candidate does not like cold, yet your company is based in northern Alaska, there is probably little need to continue the interview process.

Technical Questions
I do not suggest jumping immediately into complex technical questions, no matter what job you are interviewing a candidate for. I suggest rather working up to the point they appear to be at from their resume. This means checking for the base knowledge and experience, this is a good method to see how the candidate responds. Do they seem passionate about the work? Do they speak more about knowledge or experiences? Do they speak about where they go to stay current on the industry?

One of my favorite interview tactics is to ask a technical question about a problem we recently encountered. This is a great opportunity to see not only the candidates thought process, but also what level of detail they go into for solving the problem and what tools they would use to approach the problem. Continuing to ask questions about the problem will also show a lot about how they respond under pressure and how they communicate to other team members in the face of a stressful situation.

Technical questions also provide a good basis for seeing what knowledge the candidate has about the industry as a whole. Do they understand the benefits of one vendor or another? Or do they focus on using what they know and are comfortable with?

Domain/Vertical Knowledge
Most information technology professionals are specialized, that is, they work in a specific industry; these can include Financial Services, Oil & Gas, High Performance Computing, etc. Each of these domains has specific tools and applications, as well as industry accepted methods of accomplishing tasks.

When interviewing it is important to decide up front if you are looking for a candidate that is experienced in your specific domain, or if you are looking for a strong IT candidate that could bring an outside perspective to your organization. This will drive what questions you ask, as well as where you go to look for potential candidates to fill the position.


Interviewing is an art, it takes time to develop a process that works for you and your organization. Starting with hard technical questions will not help you determine if a candidate is qualified, there is much more to how a candidate will succeed or fail in your organization. You must treat all interviews as 2-way streets; this allows both parties to get an accurate picture all all aspects of the position including location, expectations, team dynamics and technical knowledge as well as desire to learn and evolve.

In my mind, technical skills are secondary for today's information technology positions. More important then all the technical knowledge in the world is how well the candidate communicates with team members, responds under pressure, understands trade offs and benefits analysis and ultimately the candidates desire to learn and grow and IT evolves. No matter how technical someone is, if they can not get along with them team they will be a detriment to the organization.

Talent Management in Information Technology

The Information Technology (IT) sector is evolving very rapidly, over time it has developed a reputation as a high stress career field, with low personal rewards, little chance of advancement and ultimately an area of low morale. While this is not the case within all IT departments, it has become a common perception of the industry as a whole. This perception has been driven by a variety of factors including fewer staff available to complete tasks and the use of outsourcing for lower cost resources. We as an industry need to take more ownership of our staff and skills and work to develop them internally in a way that provides companies maximum value from their IT organization, while ensuring staff are successful, stable and ultimately happy with their work environments.

First, we should define the “standard” IT employee, and I am not talking about the system administrator that works all hours of the night while drinking Red Bull. I am talking about those traits that make IT staff want to work in the technology field. While this description will not cover all IT employees, there are certain traits that stand out more in the IT space then other professions:

• Curious by nature

• Detail focused

• Quest for understanding

• Prefer the technology to the business

• Like to build things

• Opinionated

• Seeking Recognition

We now have a better understanding of what IT staff do and expect by nature. Lets explore the primary drivers that go into their ability to focus on their job and deliver successfully to the companies bottom line. This is by no means an exhaustive list, but in my experience these are the top items within an IT department that contribute to employees truly enjoying their job; employees just staying for the paycheck versus employees that are looking to move on to other opportunities.

Pay – There is a common misconception in IT that folks will work at a given company because it is interesting and exciting work, but this is only true to a point. We all have bills to pay for rent, utilities, food, entertainment, and student loans. An employee paid market average for a given area will often stay at a company if the work is exciting. But no level of excitement and interest is enough to make up for below average pay when an individual is struggling to pay their bills. Companies should strive to ensure their salaries are consistent with the local market for the level of skills an employee is utilizing at work each day.

There are also other costs that go directly with pay. It is very expensive to loose an employee, they take lots of company experience and knowledge with them. Transferring that knowledge to a new employee is costly in the time they are learning the business, they are contributing less to the bottom line. Companies should always evaluate the cost of raises for staff to ensure they stay at the area average to the cost of loosing one employee and having to hire another.

Opportunities – Staff within IT organizations are curious by nature, with that trait they constantly want to expand their knowledge and experience. Some of the most successful staff I have worked with in IT move on a regular basis, not always up in the corporate ladder, but most often laterally to other jobs that are of interest to them. This provides them a benefit of increasing their skill sets, and provides a benefit to the company because corporate knowledge is not lost when an individual moves within the company. IT staff should be provided opportunities to move both within the IT department and within the company. This movement and change of jobs is often what IT staff need to ensure they do not get burned out while allowing them to stay engaged with their jobs.

Interesting Work – Very few people are content doing the same activity every day. Because of the general trait for IT staff being on a constant quest for understanding, most IT staff are always looking for new and exciting projects. It allows them to be creative and develop new solutions to the problem. Sadly, there is always going to be some tasks that are more interesting then others within IT departments. Effort should be used to ensure that any less then desirable tasks be evenly spread across available team members, and that team members understand that while they may have been assigned a less then desirable project, so were their teammates.

Staff should be encouraged to not only take on projects assigned to them, but to come forward with ideas they have for improvement within the organization. This encourages all team members to have a stake in the organization and feel ownership of not only their projects, but other tasks that they may see a need for completing.

Flexibility – Employees appreciate when their management allows them the flexibility to work when they are most productive. I am not necessarily talking about allowing staff to work in their pajamas from home, but more referring to ensuring employees do not feel tied down with a specific schedule that causes them less productivity.

Working from home has become very common in many organizations, especially IT. I believe that a lot of organizations have taken it a step too far and staff are beginning to feel the isolation of working by themselves each and every day. I believe that staff should be provided the tools and flexibility that if they choose to work remote for an afternoon, that should be allowed. I believe that the majority of a 40-hour work week should be spent in the office, it encourages staff to communicate with their coworkers, take a vested ownership in the daily operations of the business and ensures the company develops a culture of it's own.

No two people are alike when it comes to sleeping schedules. This has a very negative effect when employees are asked to begin their day at a time that is not natural for them. I believe that staff should be given the flexibility to arrive and begin their day when they will be most productive. This does not mean everyone should sleep in until noon and begin work at 1PM. It does mean that should a staff member prefer to work later in the evening because that is when they are most productive, the company should encourage this behavior.

Ultimately an organization is only as strong as the communication between it's team members. Above I mentioned that working from home all the time is a suboptimal choice; the primary reason for that belief is that communication can be challenging when everyone is so spread out. The ability to quickly gather team members in the office and discuss a topic can ensure minimal time is wasted when a decision must be made. I believe that office hours, also called core hours, is an optimal method to ensure staff have the flexibility to work remotely, while encouraging team communication. By having all staff in the office for certain set periods, often 10-12AM and 2-4PM, it ensures that if a staff member is needed for a discussion, they can be found. This use of core hours, I believe, provides a good balance of allowing flexibility for staff and ensuring a solid team dynamic takes hold for the team.

Work Space – One common perception within IT over the years has been that putting staff in shared space will allow better collaboration. The challenge is that very little IT work is collaborative in nature, the bulk of the work that must be accomplished is individuals working on their pieces of a project, and this type of activity requires that staff be able to focus. Shared spaces have a lot of benefits in terms of quick access to others, but at the cost of decreased focus due to noise and other distractions. I believe that companies need two primary types of space available for their IT staff, private offices to allow for focus and concentration, and shared collaboration areas to allow for quick meetings and discussions.

Private office space allows staff to have an area that is their own to focus on their work and not be distracted by outside noise, phone calls, hall way discussions or projects. Each staff member should have an office that allows them to close the door and focus free of distractions.

Common areas should be available to encourage team discussions and impromptu meetings. Very rarely can a decision be made faster then by pulling the team together for a quick discussion in the hallway. These common areas within an office space should have enough white board space that notes can be kept on any design ideas or other notes from the discussion. These common areas will also encourage inclusion of all project members, and not just a subset that may discuss the matter in a private office or on a conference call.

Loyalty – Companies often expect a certain level of loyalty from all staff, but do not necessarily show that level of loyalty back to their staff. Having the CEO walk around at the company holiday party and shake hands only goes so far to telling employees that they are valued. I encourage all managers within IT to regularly call out their accomplishments of their staff to the rest of the organization. IT staff strive for recognition, it is what encourages them to do their best every day. When managers publicly acknowledge a job well done it tells the employee and all their coworkers that the effort and work are appreciated.

Now that we have those out of the way, lets explore the deep dark truth of IT. Even if a company does each of those perfectly, some staff are going to leave. This is just the nature of the business. No matter how hard a company and it's managers try, there will always be staff that are looking for something that the company can not provide. When this case occurs, and it will, it is important that the employee and company both act as professional as possible. There is an old saying in HR, “don't burn your bridges.” That applies to both the company and the individual. IT is such a rapidly evolving industry that even if the match between a company and employee is not correct now, it is very possible that a match will be made down the road after either the company or the individual evolve. “Don't burn your bridges” applies to both the employee and the employer, even if a staff member leaves, they still have institutional knowledge that could be of value down the road.

Someone I have worked with many times over the years has a very clear way to sum up the relationship between employee and employer, “Pay me well, Treat me well, Wish me well.” Meaning that if you pay your employees fairly and treat them wonderfully they will do quality work, if you pay them exceptionally well and have higher stress levels they will do quality work, but should an employee not be treated fairly and is not being paid well enough to compensate, the company should “Wish them well” in new opportunities.

Understanding Lustre Internals

Lustre can be a complex package to manage and understand. The folks at ORNL, with assistance from the Lustre Center of Excellence have put out a wonderful paper on Understanding Lustre Internals.

I recommend that all Lustre administrators read it, it is very useful information for understanding how all the Lustre pieces plug together.

"Cloud" and HPC?, Huh?

I have tried for the most part to not post on this phenomenon known as "cloud computing." "Cloud" is still evolving and as such has many different meanings. The reason this whitepaper caught my attention is it's attempt at connecting high performance computing (HPC) with "cloud computing." The way I see it, "cloud" is still more of an evolving idea then a true product. True, many companies are offering "cloud" products, but the standards are still evolving, as is the true meaning of "cloud computing."

In my mind "cloud" is the next logical evolution of computing - better resource management through enabling applications to better communicate with their supporting infrastructures (servers, storage, network, cpu and memory resources) to allow applications to have the intelligence to scale up and down based on demand. "Cloud Computing" also has a valid connection to outsourcing in the sense that shared infrastructures will at some point over take privately managed information technology (IT) infrastrucures that are common today.

There are several points about the above listed whitepaper from UnivaUD that caught my attention:

• MPI was only mentioned once. The Message Passing Interface (MPI) is the standard on which most HPC applications and platforms are built. For a paper to truly look at the potential of outsourcing HPC to a "cloud" environment, an indepth review of MPI will need to be done to ensure the proper updates are made to handle the additional physical layer errors that could occur in a shared environment, as well as the added challenges of communication in an unknown environment.
  
• There was very little mention of the actual applications that are common in HPC. Applications like Fluent, NAMD, NWChem, Gaussian, and FFTW are commonly used on clusters built in house to meet the specific needs of a given community. Moving those applications from these small, in-house envirronments will take time and review to ensure they are able to scale in shared environments, as well as properly handle the increased variation possible in hardware and configurations.
  
• There was no mention of parallel file systems. This is a fundamental requirement of modern HPC environments. To truly move common HPC environments into the "cloud" a solution will be needed for data management and transfer at the high speeds required of todays applications.

In short, the above linked whitepaper is common of what I am seeing in the "cloud" space; lots of talk of the possible benefits around the use of shared environemnts. What we need to stop doing as a community is trying to associate all things IT with "cloud." I have no doubt that in time we will evolve to more use of shared resources - this has been occuring for quite a while with the migration to larger clusters within universities and national laboratories, as well as the ongoing outsourcing of email and specific applications - but as a community we need to ensure that each time we change how we do things for a given area of IT it is with specific goals in mind. Without those clearly defined goals we will not know if we were successful.

As time allows I hope to explore the above issues, particularly looking at alternatives for parallel file systems in environments that may have varying latency, and are distributed over various data centers.